[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:06 dist-test-slave-w4v5 krb5kdc[28103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:06 dist-test-slave-w4v5 krb5kdc[28103](info): set up 2 sockets
Aug 12 07:00:06 dist-test-slave-w4v5 krb5kdc[28103](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 07:00:08.204320 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.374s user 0.010s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:08 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:08.336570 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:45245
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:45245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:08.459311 28129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:08.459611 28129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:08.459702 28129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:08.463779 28129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:08.463860 28129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:08.463898 28129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:08.463954 28129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:08.463989 28129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:08.469337 28129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:45245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:45245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:08.470778 28129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:08.472085 28129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:08.484206 28138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:08.484344 28139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:08.484488 28142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:08.484303 28129 server_base.cc:1034] running on GCE node
I20260812 07:00:08.485234 28129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:08.486914 28129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:08.488085 28129 hybrid_clock.cc:648] HybridClock initialized: now 1786518008488063 us; error 64 us; skew 500 ppm
Aug 12 07:00:08 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:08.491191 28129 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:08.492241 28129 webserver.cc:460] Webserver started at http://127.27.110.254:34207/ using document root <none> and password file <none>
I20260812 07:00:08.493069 28129 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:08.493175 28129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:08.493588 28129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:08.496289 28129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "51b93786a31946269d36073aba78a448"
format_stamp: "Formatted at 2026-08-12 07:00:08 on dist-test-slave-w4v5"
server_key: "7f7f5f07818bda02e8544f0f9f69914f"
server_key_iv: "354df664ea6cc2ef76bc0078c14305dd"
server_key_version: "encryptionkey@0"
I20260812 07:00:08.496886 28129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "51b93786a31946269d36073aba78a448"
format_stamp: "Formatted at 2026-08-12 07:00:08 on dist-test-slave-w4v5"
server_key: "7f7f5f07818bda02e8544f0f9f69914f"
server_key_iv: "354df664ea6cc2ef76bc0078c14305dd"
server_key_version: "encryptionkey@0"
I20260812 07:00:08.501835 28129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:08.505349 28161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:08.506963 28129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:08.507264 28129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "51b93786a31946269d36073aba78a448"
format_stamp: "Formatted at 2026-08-12 07:00:08 on dist-test-slave-w4v5"
server_key: "7f7f5f07818bda02e8544f0f9f69914f"
server_key_iv: "354df664ea6cc2ef76bc0078c14305dd"
server_key_version: "encryptionkey@0"
I20260812 07:00:08.507395 28129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:08.529284 28129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:08.532733 28129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:08.533000 28129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:08.542048 28129 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:45245
I20260812 07:00:08.542109 28258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:45245 every 8 connection(s)
I20260812 07:00:08.543385 28129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:08.545293 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28129
I20260812 07:00:08.545518 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:08.547873 28091 external_mini_cluster.cc:1428] Setting key 5555752daba1f028c27e6525b543bb65
I20260812 07:00:08.549546 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:08.560685 28260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448: Bootstrap starting.
I20260812 07:00:08.565368 28260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:08.567113 28260 log.cc:824] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:08.570257 28260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448: No bootstrap required, opened a new log
Aug 12 07:00:08 dist-test-slave-w4v5 krb5kdc[28103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:08.577060 28260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b93786a31946269d36073aba78a448" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 45245 } }
I20260812 07:00:08.577414 28260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:08.577528 28260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b93786a31946269d36073aba78a448, State: Initialized, Role: FOLLOWER
I20260812 07:00:08.578450 28260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b93786a31946269d36073aba78a448" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 45245 } }
I20260812 07:00:08.578709 28260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:08.578859 28260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:08.579020 28260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:08.580389 28260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b93786a31946269d36073aba78a448" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 45245 } }
I20260812 07:00:08.581097 28260 leader_election.cc:302] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51b93786a31946269d36073aba78a448; no voters:
I20260812 07:00:08.581560 28260 leader_election.cc:288] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:08.581732 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:08.553931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.27.110.254:45245)
0812 07:00:08.554466 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:08.554487 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:08.555819 (+ 1332us) server_negotiation.cc:407] Connection header received
0812 07:00:08.560795 (+ 4976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:08.560821 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:08.561205 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:08.561462 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:08.565307 (+ 3845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:08.566466 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:08.568288 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:08.568786 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:08.568906 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:08.574756 (+ 5850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:08.574788 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:08.574801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:08.574827 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:08.577438 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:08.578228 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:08.578233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:08.578237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:08.578346 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:08.578786 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:08.578790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:08.578792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:08.579405 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:08.579573 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:08.580332 (+ 759us) server_negotiation.cc:299] Negotiation successful
0812 07:00:08.580615 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":131,"threads_started":1}
I20260812 07:00:08.581804 28275 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:08.583417 28260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:08.583498 28275 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [term 1 LEADER]: Becoming Leader. State: Replica: 51b93786a31946269d36073aba78a448, State: Running, Role: LEADER
I20260812 07:00:08.583911 28275 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b93786a31946269d36073aba78a448" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 45245 } }
I20260812 07:00:08.586643 28280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51b93786a31946269d36073aba78a448. Latest consensus state: current_term: 1 leader_uuid: "51b93786a31946269d36073aba78a448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b93786a31946269d36073aba78a448" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 45245 } } }
I20260812 07:00:08.586890 28280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:08.587869 28296 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:08.587814 28278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51b93786a31946269d36073aba78a448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51b93786a31946269d36073aba78a448" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 45245 } } }
I20260812 07:00:08.587990 28278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:08.592919 28296 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:08.602252 28296 catalog_manager.cc:1347] Generated new cluster ID: 363e13476b664641be87ea26a17fa63e
I20260812 07:00:08.602334 28296 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:08.612504 28296 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:08.613907 28296 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:08.624603 28296 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51b93786a31946269d36073aba78a448: Generated new TSK 0
I20260812 07:00:08.625705 28296 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:08.664134 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:45245
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:08.787902 28314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:08.788168 28314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:08.788214 28314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:08.792230 28314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:08.792291 28314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:08.792389 28314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:08.797739 28314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:45245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:08.798889 28314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:08.799968 28314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:08.810595 28327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:08.810602 28325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:08.810595 28331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:08.810844 28314 server_base.cc:1034] running on GCE node
I20260812 07:00:08.811383 28314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:08.812098 28314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:08.813294 28314 hybrid_clock.cc:648] HybridClock initialized: now 1786518008813261 us; error 46 us; skew 500 ppm
Aug 12 07:00:08 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:08.816066 28314 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:08.816952 28314 webserver.cc:460] Webserver started at http://127.27.110.193:38821/ using document root <none> and password file <none>
I20260812 07:00:08.817574 28314 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:08.817667 28314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:08.817873 28314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:08.820001 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "403bffe487a044cf9db99bb26ad71fc2"
format_stamp: "Formatted at 2026-08-12 07:00:08 on dist-test-slave-w4v5"
server_key: "0efe8e44ae7786042c17a490b1a6d4bb"
server_key_iv: "02de5b020f6159ded1ec450e93b0235b"
server_key_version: "encryptionkey@0"
I20260812 07:00:08.820564 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "403bffe487a044cf9db99bb26ad71fc2"
format_stamp: "Formatted at 2026-08-12 07:00:08 on dist-test-slave-w4v5"
server_key: "0efe8e44ae7786042c17a490b1a6d4bb"
server_key_iv: "02de5b020f6159ded1ec450e93b0235b"
server_key_version: "encryptionkey@0"
I20260812 07:00:08.824319 28314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:00:08.826643 28341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:08.827967 28314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:08.828097 28314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "403bffe487a044cf9db99bb26ad71fc2"
format_stamp: "Formatted at 2026-08-12 07:00:08 on dist-test-slave-w4v5"
server_key: "0efe8e44ae7786042c17a490b1a6d4bb"
server_key_iv: "02de5b020f6159ded1ec450e93b0235b"
server_key_version: "encryptionkey@0"
I20260812 07:00:08.828218 28314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:08.862488 28314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:08.865396 28314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:08.865597 28314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:08.866294 28314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:08.867424 28314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:08.867501 28314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:08.867573 28314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:08.867621 28314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:08.877146 28314 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:44753
I20260812 07:00:08.877177 28570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:44753 every 8 connection(s)
I20260812 07:00:08.877996 28314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:08.881870 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28314
I20260812 07:00:08.881980 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:08.882258 28091 external_mini_cluster.cc:1428] Setting key 24d4a46e845dac2e063d8eba9b8cfe91
Aug 12 07:00:08 dist-test-slave-w4v5 krb5kdc[28103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518008, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:08.893944 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:08.880197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:34623 (local address 127.27.110.254:45245)
0812 07:00:08.880327 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:08.880330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:08.881415 (+ 1085us) server_negotiation.cc:407] Connection header received
0812 07:00:08.882657 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:08.882661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:08.882744 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:08.882841 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:08.884384 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:08.884948 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:08.886250 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:08.886668 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:08.886790 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:08.889815 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:08.889819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:08.889821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:08.889846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:08.891520 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:08.892160 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:08.892163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:08.892164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:08.892247 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:08.892693 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:08.892699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:08.892701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:08.892880 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:08.892958 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:08.893653 (+ 695us) server_negotiation.cc:299] Negotiation successful
0812 07:00:08.893763 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:00:08.894784 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:08.880629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:45245 (local address 127.27.110.193:34623)
0812 07:00:08.881262 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:08.881302 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:08.882390 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:08.883051 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:08.883060 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:08.883499 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:08.884232 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:08.884254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:08.885135 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:08.885138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:08.886082 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:08.886092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:08.886855 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:08.886859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:08.887007 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:08.887704 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:08.887720 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:08.889642 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:08.891642 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:08.891649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:08.891664 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:08.892044 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:08.892365 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:08.892369 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:08.892371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:08.892565 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:08.892998 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:08.893007 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:08.893358 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 07:00:08.893589 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 07:00:08.893942 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":113,"threads_started":1}
I20260812 07:00:08.896004 28572 heartbeater.cc:347] Connected to a master server at 127.27.110.254:45245
I20260812 07:00:08.896255 28572 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:08.896827 28572 heartbeater.cc:511] Master 127.27.110.254:45245 requested a full tablet report, sending...
I20260812 07:00:08.898968 28191 ts_manager.cc:194] Registered new tserver with Master: 403bffe487a044cf9db99bb26ad71fc2 (127.27.110.193:44753)
I20260812 07:00:08.900460 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:34623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:08.912572 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:45245
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:09.025111 28582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:09.025406 28582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:09.025477 28582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:09.029521 28582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:09.029608 28582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:09.029704 28582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:09.035068 28582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:45245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:09.036206 28582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:09.037257 28582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:09.047115 28591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:09.047111 28588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:09.047137 28587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:09.047385 28582 server_base.cc:1034] running on GCE node
I20260812 07:00:09.047845 28582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:09.048504 28582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:09.049705 28582 hybrid_clock.cc:648] HybridClock initialized: now 1786518009049686 us; error 26 us; skew 500 ppm
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:09.052578 28582 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:09.053483 28582 webserver.cc:460] Webserver started at http://127.27.110.194:37137/ using document root <none> and password file <none>
I20260812 07:00:09.054050 28582 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:09.054147 28582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:09.054387 28582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:09.056454 28582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "a4ae63de3c4945a79264e741fed2e745"
format_stamp: "Formatted at 2026-08-12 07:00:09 on dist-test-slave-w4v5"
server_key: "1b7d51d63643e16560daf0cdb23f7720"
server_key_iv: "a4c8af0c3b3410634afcd3d3c294a414"
server_key_version: "encryptionkey@0"
I20260812 07:00:09.057101 28582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "a4ae63de3c4945a79264e741fed2e745"
format_stamp: "Formatted at 2026-08-12 07:00:09 on dist-test-slave-w4v5"
server_key: "1b7d51d63643e16560daf0cdb23f7720"
server_key_iv: "a4c8af0c3b3410634afcd3d3c294a414"
server_key_version: "encryptionkey@0"
I20260812 07:00:09.060834 28582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260812 07:00:09.063242 28603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:09.064399 28582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:09.064548 28582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "a4ae63de3c4945a79264e741fed2e745"
format_stamp: "Formatted at 2026-08-12 07:00:09 on dist-test-slave-w4v5"
server_key: "1b7d51d63643e16560daf0cdb23f7720"
server_key_iv: "a4c8af0c3b3410634afcd3d3c294a414"
server_key_version: "encryptionkey@0"
I20260812 07:00:09.064661 28582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:09.098099 28582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:09.101125 28582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:09.101346 28582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:09.101989 28582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:09.102836 28582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:09.102909 28582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:09.102977 28582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:09.103021 28582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:09.112532 28582 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:39615
I20260812 07:00:09.112560 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:39615 every 8 connection(s)
I20260812 07:00:09.113453 28582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:09.118872 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28582
I20260812 07:00:09.118988 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:09.119277 28091 external_mini_cluster.cc:1428] Setting key 31577bfc1c69cb4f4af0dae798155d0a
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:09.128942 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:09.115226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:42945 (local address 127.27.110.254:45245)
0812 07:00:09.115370 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:09.115374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:09.116436 (+ 1062us) server_negotiation.cc:407] Connection header received
0812 07:00:09.117277 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:09.117280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:09.117325 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:09.117400 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:09.118706 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.119390 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.120880 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.121109 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.121199 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:09.124498 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:09.124505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:09.124511 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:09.124547 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:09.126567 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.127186 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.127192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.127197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.127298 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.127665 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.127670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.127674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.127987 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:09.128115 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:09.128595 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 07:00:09.128737 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:00:09.129652 28822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:09.115643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:45245 (local address 127.27.110.194:42945)
0812 07:00:09.116291 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:09.116334 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:09.117101 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:09.117592 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:09.117599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:09.117960 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:09.118542 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:09.118556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.119531 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.119534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:09.120745 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:09.120756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.121274 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.121281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:09.121509 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:09.122165 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:09.122191 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:09.124309 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:09.126702 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:09.126710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:09.126729 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:09.127029 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:09.127409 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:09.127412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:09.127413 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:09.127552 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:09.128092 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:09.128098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:09.128329 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 07:00:09.128482 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 07:00:09.128798 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":107,"threads_started":1}
I20260812 07:00:09.130949 28818 heartbeater.cc:347] Connected to a master server at 127.27.110.254:45245
I20260812 07:00:09.131294 28818 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:09.131896 28818 heartbeater.cc:511] Master 127.27.110.254:45245 requested a full tablet report, sending...
I20260812 07:00:09.133047 28191 ts_manager.cc:194] Registered new tserver with Master: a4ae63de3c4945a79264e741fed2e745 (127.27.110.194:39615)
I20260812 07:00:09.133726 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:42945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:09.149981 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:45245
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:09.264495 28826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:09.264781 28826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:09.264858 28826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:09.268929 28826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:09.269011 28826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:09.269102 28826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:09.274271 28826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:45245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:09.275441 28826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:09.276489 28826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:09.286576 28836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:09.286580 28846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:09.286770 28826 server_base.cc:1034] running on GCE node
W20260812 07:00:09.286924 28840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:09.287381 28826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:09.288057 28826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:09.289403 28826 hybrid_clock.cc:648] HybridClock initialized: now 1786518009289375 us; error 44 us; skew 500 ppm
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:09.292743 28826 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:09.293674 28826 webserver.cc:460] Webserver started at http://127.27.110.195:34077/ using document root <none> and password file <none>
I20260812 07:00:09.294260 28826 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:09.294356 28826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:09.294597 28826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:09.296773 28826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "5cde897bd85c4c45a2ae52ef3a764f1d"
format_stamp: "Formatted at 2026-08-12 07:00:09 on dist-test-slave-w4v5"
server_key: "edc3cb19df78b66d4aa063f4e836b66e"
server_key_iv: "0e91b1c64541051cf4886453e72514be"
server_key_version: "encryptionkey@0"
I20260812 07:00:09.297370 28826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "5cde897bd85c4c45a2ae52ef3a764f1d"
format_stamp: "Formatted at 2026-08-12 07:00:09 on dist-test-slave-w4v5"
server_key: "edc3cb19df78b66d4aa063f4e836b66e"
server_key_iv: "0e91b1c64541051cf4886453e72514be"
server_key_version: "encryptionkey@0"
I20260812 07:00:09.301241 28826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:09.304042 28860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:09.305251 28826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:09.305392 28826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "5cde897bd85c4c45a2ae52ef3a764f1d"
format_stamp: "Formatted at 2026-08-12 07:00:09 on dist-test-slave-w4v5"
server_key: "edc3cb19df78b66d4aa063f4e836b66e"
server_key_iv: "0e91b1c64541051cf4886453e72514be"
server_key_version: "encryptionkey@0"
I20260812 07:00:09.305508 28826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:09.319353 28826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:09.322177 28826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:09.322384 28826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:09.322995 28826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:09.323863 28826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:09.323937 28826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:09.324011 28826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:09.324056 28826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:09.333389 28826 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44493
I20260812 07:00:09.333416 29065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44493 every 8 connection(s)
I20260812 07:00:09.334266 28826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:09.336233 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28826
I20260812 07:00:09.336359 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:09.336647 28091 external_mini_cluster.cc:1428] Setting key c7e9e133f5529c47608a49dec21c9c44
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:09.348073 28264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:09.336137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:35443 (local address 127.27.110.254:45245)
0812 07:00:09.336270 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:09.336274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:09.337598 (+ 1324us) server_negotiation.cc:407] Connection header received
0812 07:00:09.338786 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:09.338790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:09.338839 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:09.338914 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:09.340233 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.340809 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.341921 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.342152 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.342241 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:09.344721 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:09.344724 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:09.344726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:09.344750 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:09.345985 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.346499 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.346503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.346505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.346591 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.346952 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.346956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.346958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.347188 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:09.347297 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:09.347777 (+ 480us) server_negotiation.cc:299] Negotiation successful
0812 07:00:09.347901 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:00:09.348822 29069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:09.336642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:45245 (local address 127.27.110.195:35443)
0812 07:00:09.337437 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:09.337479 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:09.338510 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:09.339158 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:09.339168 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:09.339620 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:09.340097 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:09.340110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.340926 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.340929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:09.341792 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:09.341801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.342293 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.342296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:09.342435 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:09.343001 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:09.343018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:09.344544 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:09.346107 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:09.346120 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:09.346134 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:09.346389 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:09.346696 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:09.346699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:09.346702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:09.346851 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:09.347297 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:09.347305 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:09.347536 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 07:00:09.347722 (+ 186us) client_negotiation.cc:240] Negotiation successful
0812 07:00:09.348025 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":149,"threads_started":1}
I20260812 07:00:09.349876 29066 heartbeater.cc:347] Connected to a master server at 127.27.110.254:45245
I20260812 07:00:09.350132 29066 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:09.350732 29066 heartbeater.cc:511] Master 127.27.110.254:45245 requested a full tablet report, sending...
I20260812 07:00:09.351801 28191 ts_manager.cc:194] Registered new tserver with Master: 5cde897bd85c4c45a2ae52ef3a764f1d (127.27.110.195:44493)
I20260812 07:00:09.352349 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:35443
I20260812 07:00:09.363143 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:09.391239 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:09.381559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39636 (local address 127.27.110.193:44753)
0812 07:00:09.381899 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:09.381914 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:09.381943 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:00:09.382002 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:09.382008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:09.382199 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:09.382330 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:09.382971 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.383996 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.384862 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.385331 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.385428 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:09.387941 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:09.387951 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:09.387964 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:09.387989 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:09.389521 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.389963 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.389967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.389968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.390048 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.390296 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.390298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.390299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.390546 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:09.390731 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:09.390895 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 07:00:09.391021 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
W20260812 07:00:09.392369 28403 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39636
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[28103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:09.420382 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:09.412040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.27.110.193:44753)
0812 07:00:09.412199 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:09.412203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:09.412340 (+ 137us) server_negotiation.cc:407] Connection header received
0812 07:00:09.412442 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:09.412445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:09.412488 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:09.412609 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:09.413290 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.413821 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.414605 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:09.414832 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:09.414907 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:09.417292 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:09.417296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:09.417298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:09.417321 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:09.418778 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.419305 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.419307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.419308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.419391 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:09.419612 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:09.419614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:09.419615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:09.419767 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:09.419862 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:09.420059 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 07:00:09.420160 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:00:09.421627 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28314
I20260812 07:00:09.437237 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28582
I20260812 07:00:09.448976 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28826
I20260812 07:00:09.462337 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28129
2026-08-12T07:00:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2674 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[29099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[29099](info): set up 2 sockets
Aug 12 07:00:09 dist-test-slave-w4v5 krb5kdc[29099](info): commencing operation
krb5kdc: starting...
W20260812 07:00:10.771905 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.280s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:10 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:10.898838 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39041
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:11.012406 29118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:11.012691 29118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:11.012732 29118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:11.016784 29118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:11.016844 29118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:11.016863 29118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:11.016880 29118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:11.016927 29118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:11.022205 29118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:11.023350 29118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:11.024375 29118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:00:11.034241 29118 server_base.cc:1034] running on GCE node
W20260812 07:00:11.034173 29133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.034152 29131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.034137 29140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:11.035140 29118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:11.036325 29118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:11.037688 29118 hybrid_clock.cc:648] HybridClock initialized: now 1786518011037661 us; error 46 us; skew 500 ppm
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:11.040658 29118 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:11.041543 29118 webserver.cc:460] Webserver started at http://127.27.110.254:37633/ using document root <none> and password file <none>
I20260812 07:00:11.042109 29118 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:11.042202 29118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:11.042439 29118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:11.044564 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "b7994a39b3944309b0f7d133de4d3b32"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "d048fd710132045857968db97325d9a7"
server_key_iv: "d035d5840e8359067bf96dc7d1cb463b"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.045194 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "b7994a39b3944309b0f7d133de4d3b32"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "d048fd710132045857968db97325d9a7"
server_key_iv: "d035d5840e8359067bf96dc7d1cb463b"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.048983 29118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:00:11.051358 29159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.052556 29118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:11.052706 29118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "b7994a39b3944309b0f7d133de4d3b32"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "d048fd710132045857968db97325d9a7"
server_key_iv: "d035d5840e8359067bf96dc7d1cb463b"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.052819 29118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:11.069561 29118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:11.072814 29118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:11.073024 29118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:11.080127 29118 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39041
I20260812 07:00:11.080156 29260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39041 every 8 connection(s)
I20260812 07:00:11.081100 29118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:11.083803 29261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:11.086074 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29118
I20260812 07:00:11.086172 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:11.086437 28091 external_mini_cluster.cc:1428] Setting key fa62d75b2b182e727dbca793590ff38d
I20260812 07:00:11.089920 29261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32: Bootstrap starting.
I20260812 07:00:11.092549 29261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:11.093364 29261 log.cc:824] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:11.095638 29261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32: No bootstrap required, opened a new log
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:11.099790 29261 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7994a39b3944309b0f7d133de4d3b32" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39041 } }
I20260812 07:00:11.100076 29261 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:11.100152 29261 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7994a39b3944309b0f7d133de4d3b32, State: Initialized, Role: FOLLOWER
I20260812 07:00:11.100623 29261 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7994a39b3944309b0f7d133de4d3b32" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39041 } }
I20260812 07:00:11.100807 29261 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:11.100931 29261 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:11.101065 29261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:11.102181 29261 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7994a39b3944309b0f7d133de4d3b32" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39041 } }
I20260812 07:00:11.102543 29261 leader_election.cc:302] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7994a39b3944309b0f7d133de4d3b32; no voters:
I20260812 07:00:11.102870 29261 leader_election.cc:288] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:11.103001 29271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:11.103276 29271 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [term 1 LEADER]: Becoming Leader. State: Replica: b7994a39b3944309b0f7d133de4d3b32, State: Running, Role: LEADER
I20260812 07:00:11.103636 29271 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7994a39b3944309b0f7d133de4d3b32" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39041 } }
I20260812 07:00:11.103952 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.087884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.27.110.254:39041)
0812 07:00:11.088298 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.088316 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.088360 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:00:11.089192 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.089215 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.089604 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.089978 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:11.090694 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.092230 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.093172 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.093905 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.094084 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.097738 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:11.097744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:11.097761 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:11.097796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:11.100782 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.101276 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.101285 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.101294 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.101430 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.101693 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.101696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.101698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.102225 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:11.102415 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.102682 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.102981 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":111,"threads_started":1}
I20260812 07:00:11.104193 29261 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:11.107036 29274 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7994a39b3944309b0f7d133de4d3b32. Latest consensus state: current_term: 1 leader_uuid: "b7994a39b3944309b0f7d133de4d3b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7994a39b3944309b0f7d133de4d3b32" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39041 } } }
I20260812 07:00:11.107582 29274 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:11.107334 29272 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7994a39b3944309b0f7d133de4d3b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7994a39b3944309b0f7d133de4d3b32" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39041 } } }
I20260812 07:00:11.107832 29272 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32 [sys.catalog]: This master's current role is: LEADER
W20260812 07:00:11.109570 29296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:11.109690 29296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:11.109760 29297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:11.110632 29297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:11.116114 29297 catalog_manager.cc:1347] Generated new cluster ID: 849d20047733473ab0650edb933639ae
I20260812 07:00:11.116214 29297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:11.125026 29297 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:11.125878 29297 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:11.134238 29297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7994a39b3944309b0f7d133de4d3b32: Generated new TSK 0
I20260812 07:00:11.134929 29297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:11.170065 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39041
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:11.298480 29315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:11.298756 29315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:11.298810 29315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:11.302752 29315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:11.302809 29315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:11.302866 29315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:11.308173 29315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:11.309151 29315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:11.310101 29315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:11.319769 29332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.319782 29326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.319787 29325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:11.320158 29315 server_base.cc:1034] running on GCE node
I20260812 07:00:11.320541 29315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:11.321163 29315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:11.322369 29315 hybrid_clock.cc:648] HybridClock initialized: now 1786518011322301 us; error 76 us; skew 500 ppm
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:11.325291 29315 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:11.326071 29315 webserver.cc:460] Webserver started at http://127.27.110.193:38131/ using document root <none> and password file <none>
I20260812 07:00:11.326575 29315 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:11.326637 29315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:11.326794 29315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:11.328789 29315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "7f58d12cab9048d591f8d7c97522faf0"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "200a55d962431bbee4bf610388eb1326"
server_key_iv: "7f979a5e4565610dbcf6d05027b8acb1"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.329296 29315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "7f58d12cab9048d591f8d7c97522faf0"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "200a55d962431bbee4bf610388eb1326"
server_key_iv: "7f979a5e4565610dbcf6d05027b8acb1"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.332918 29315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:00:11.335237 29344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.336426 29315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:11.336519 29315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "7f58d12cab9048d591f8d7c97522faf0"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "200a55d962431bbee4bf610388eb1326"
server_key_iv: "7f979a5e4565610dbcf6d05027b8acb1"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.336647 29315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:11.366413 29315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:11.369691 29315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:11.369911 29315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:11.370572 29315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:11.371471 29315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:11.371546 29315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.371619 29315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:11.371672 29315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.381150 29315 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:36345
I20260812 07:00:11.381165 29566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:36345 every 8 connection(s)
I20260812 07:00:11.382105 29315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:11.387806 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29315
I20260812 07:00:11.387926 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:11.388173 28091 external_mini_cluster.cc:1428] Setting key 0a207ff348693194ce954b29a2c1390c
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:11.396780 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.384018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:41907 (local address 127.27.110.254:39041)
0812 07:00:11.384169 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.384173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.385042 (+ 869us) server_negotiation.cc:407] Connection header received
0812 07:00:11.386006 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.386010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.386071 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.386179 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:11.387412 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.388237 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.389644 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.389957 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.390070 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.393040 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:11.393043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:11.393045 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:11.393070 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:11.394555 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.395141 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.395146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.395148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.395242 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.395616 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.395619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.395621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.395815 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:11.395928 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.396467 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.396599 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:00:11.397620 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.384403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39041 (local address 127.27.110.193:41907)
0812 07:00:11.384883 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:11.384929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:11.385758 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:11.386370 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:11.386377 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:11.386743 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:11.387254 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.387269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.388388 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.388391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.389503 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.389514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.390185 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.390188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.390354 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.391038 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:11.391062 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:11.392858 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:11.394703 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:11.394709 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:11.394724 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:11.394998 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:11.395344 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:11.395347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:11.395349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:11.395503 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:11.395930 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:11.395938 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:11.396193 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 07:00:11.396408 (+ 215us) client_negotiation.cc:240] Negotiation successful
0812 07:00:11.396758 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:11.398986 29567 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39041
I20260812 07:00:11.399283 29567 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:11.399758 29567 heartbeater.cc:511] Master 127.27.110.254:39041 requested a full tablet report, sending...
I20260812 07:00:11.401291 29196 ts_manager.cc:194] Registered new tserver with Master: 7f58d12cab9048d591f8d7c97522faf0 (127.27.110.193:36345)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:11.402643 29196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:41907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:11.418031 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39041
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:11.531239 29584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:11.531517 29584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:11.531565 29584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:11.535681 29584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:11.535743 29584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:11.535851 29584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:11.541172 29584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:11.542258 29584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:11.543323 29584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:11.553308 29610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.553393 29607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.553323 29604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:11.553529 29584 server_base.cc:1034] running on GCE node
I20260812 07:00:11.554057 29584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:11.554729 29584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:11.555945 29584 hybrid_clock.cc:648] HybridClock initialized: now 1786518011555932 us; error 32 us; skew 500 ppm
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:11.559281 29584 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:11.560228 29584 webserver.cc:460] Webserver started at http://127.27.110.194:46611/ using document root <none> and password file <none>
I20260812 07:00:11.560842 29584 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:11.560946 29584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:11.561213 29584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:11.563478 29584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "76981da8dbde4bf09beb05fd5ace8103"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "cf7820d2ed6a147677175908a312b91b"
server_key_iv: "966984ffadc5d3e8756ba9c353228111"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.564131 29584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "76981da8dbde4bf09beb05fd5ace8103"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "cf7820d2ed6a147677175908a312b91b"
server_key_iv: "966984ffadc5d3e8756ba9c353228111"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.568123 29584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:00:11.570688 29617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.572016 29584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:11.572157 29584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "76981da8dbde4bf09beb05fd5ace8103"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "cf7820d2ed6a147677175908a312b91b"
server_key_iv: "966984ffadc5d3e8756ba9c353228111"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.572273 29584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:11.592232 29584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:11.595798 29584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:11.596014 29584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:11.596644 29584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:11.597504 29584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:11.597579 29584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.597651 29584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:11.597711 29584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.607647 29584 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:44113
I20260812 07:00:11.607726 29835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:44113 every 8 connection(s)
I20260812 07:00:11.608713 29584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:11.614997 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29584
I20260812 07:00:11.615137 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:11.615428 28091 external_mini_cluster.cc:1428] Setting key e5520af8c7403e5c5d3d732289389331
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:11.623706 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.610560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57621 (local address 127.27.110.254:39041)
0812 07:00:11.610715 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.610720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.611775 (+ 1055us) server_negotiation.cc:407] Connection header received
0812 07:00:11.612749 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.612752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.612799 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.612875 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:11.614158 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.614679 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.616360 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.616593 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.616710 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.619569 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:11.619576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:11.619580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:11.619614 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:11.621280 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.621818 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.621825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.621829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.621939 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.622321 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.622328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.622333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.622630 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:11.622747 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.623297 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.623450 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 07:00:11.624411 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.611054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39041 (local address 127.27.110.194:57621)
0812 07:00:11.611607 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:11.611655 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:11.612536 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:11.613082 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:11.613090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:11.613472 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:11.614023 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.614039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.614830 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.614837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.616211 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.616229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.616751 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.616755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.616950 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.617577 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:11.617603 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:11.619381 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:11.621409 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:11.621413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:11.621425 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:11.621695 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:11.622044 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:11.622046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:11.622048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:11.622201 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:11.622738 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:11.622743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:11.622976 (+ 233us) client_negotiation.cc:769] Sending connection context
0812 07:00:11.623204 (+ 228us) client_negotiation.cc:240] Negotiation successful
0812 07:00:11.623473 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":121,"threads_started":1}
I20260812 07:00:11.625771 29836 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39041
I20260812 07:00:11.626025 29836 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:11.626530 29836 heartbeater.cc:511] Master 127.27.110.254:39041 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:11.627621 29196 ts_manager.cc:194] Registered new tserver with Master: 76981da8dbde4bf09beb05fd5ace8103 (127.27.110.194:44113)
I20260812 07:00:11.628233 29196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:57621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:11.645828 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39041
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:11.760545 29849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:11.760799 29849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:11.760844 29849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:11.764858 29849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:11.764912 29849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:11.764969 29849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:11.770067 29849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39041
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:11.771112 29849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:11.772104 29849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:11.782466 29868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.782481 29869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:11.782445 29871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:11.782794 29849 server_base.cc:1034] running on GCE node
I20260812 07:00:11.783344 29849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:11.784003 29849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:11.785223 29849 hybrid_clock.cc:648] HybridClock initialized: now 1786518011785171 us; error 62 us; skew 500 ppm
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:11.788352 29849 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:11.789223 29849 webserver.cc:460] Webserver started at http://127.27.110.195:33681/ using document root <none> and password file <none>
I20260812 07:00:11.789819 29849 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:11.789891 29849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:11.790131 29849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:11.792356 29849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "ea2d0c15d9ea43819de6ca7876259613"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "b656aea0745891736c5a15f4d82485a6"
server_key_iv: "5d54392036f8ff80e337d4eee743aeff"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.792968 29849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "ea2d0c15d9ea43819de6ca7876259613"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "b656aea0745891736c5a15f4d82485a6"
server_key_iv: "5d54392036f8ff80e337d4eee743aeff"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.796854 29849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:11.799316 29882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.800562 29849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:11.800704 29849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "ea2d0c15d9ea43819de6ca7876259613"
format_stamp: "Formatted at 2026-08-12 07:00:11 on dist-test-slave-w4v5"
server_key: "b656aea0745891736c5a15f4d82485a6"
server_key_iv: "5d54392036f8ff80e337d4eee743aeff"
server_key_version: "encryptionkey@0"
I20260812 07:00:11.800810 29849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:11.813651 29849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:11.817003 29849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:11.817214 29849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:11.817832 29849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:11.818650 29849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:11.818698 29849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.818768 29849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:11.818811 29849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:11.828459 29849 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:36753
I20260812 07:00:11.828495 30127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:36753 every 8 connection(s)
I20260812 07:00:11.829336 29849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:11.833705 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29849
I20260812 07:00:11.833863 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:11.834128 28091 external_mini_cluster.cc:1428] Setting key 9c7c848a5e72bb5946703fdef20eaf8c
Aug 12 07:00:11 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518011, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:11.843257 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.831258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:49447 (local address 127.27.110.254:39041)
0812 07:00:11.831428 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.831433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.832315 (+ 882us) server_negotiation.cc:407] Connection header received
0812 07:00:11.833164 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.833167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.833214 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.833293 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:11.834648 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.835457 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.836708 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.836919 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.837001 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.839655 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:11.839658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:11.839661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:11.839684 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:11.841407 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.841888 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.841891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.841892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.841969 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.842270 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.842272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.842273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.842439 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:11.842514 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.842977 (+ 463us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.843124 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:00:11.844070 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.831678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39041 (local address 127.27.110.195:49447)
0812 07:00:11.832148 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:11.832191 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:11.832918 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:11.833467 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:11.833474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:11.833863 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:11.834459 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.834480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.835591 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.835594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.836597 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.836607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.837056 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.837059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.837203 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.837790 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:11.837806 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:11.839417 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:11.841534 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:11.841538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:11.841549 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:11.841798 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:11.842050 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:11.842052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:11.842058 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:11.842188 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:11.842542 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:11.842547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:11.842763 (+ 216us) client_negotiation.cc:769] Sending connection context
0812 07:00:11.842918 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 07:00:11.843231 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":109,"threads_started":1}
I20260812 07:00:11.845218 30128 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39041
I20260812 07:00:11.845464 30128 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:11.845942 30128 heartbeater.cc:511] Master 127.27.110.254:39041 requested a full tablet report, sending...
I20260812 07:00:11.847018 29196 ts_manager.cc:194] Registered new tserver with Master: ea2d0c15d9ea43819de6ca7876259613 (127.27.110.195:36753)
I20260812 07:00:11.847783 29196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:49447
I20260812 07:00:11.848045 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:00:11.859150 29265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.852248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.27.110.254:39041)
0812 07:00:11.852361 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.852365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.852523 (+ 158us) server_negotiation.cc:407] Connection header received
0812 07:00:11.852605 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.852607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.852644 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.852709 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:11.853364 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.853849 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.854627 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.854838 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.854917 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.856175 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:11.856178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:11.856179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:11.856201 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:11.857635 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.858013 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.858015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.858016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.858086 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:11.858306 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:11.858308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:11.858309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:11.858501 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:11.858602 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.858818 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.858903 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:00:11.864645 29196 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:11.867961 29196 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:11.883208 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.877352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:36753 (local address 127.0.0.1:53756)
0812 07:00:11.877780 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:11.877824 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:11.877952 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:11.879312 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:11.879319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:11.879348 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:11.879507 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.879514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.880648 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.880654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.881810 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.881819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.882614 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.882618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.882744 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.882785 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:00:11.882905 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 07:00:11.883035 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":79,"threads_started":1}
I20260812 07:00:11.883358 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.877493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.27.110.195:36753)
0812 07:00:11.878785 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.878798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.878821 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:00:11.878903 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.878909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.879142 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.879298 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:11.879647 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.880510 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.881930 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.882487 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.882635 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.882829 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.882934 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.883035 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":136,"threads_started":1}
I20260812 07:00:11.883610 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.878064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:44113 (local address 127.0.0.1:57308)
0812 07:00:11.878426 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:11.878439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:11.878525 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:11.879751 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:11.879753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:11.879771 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:11.879841 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.879847 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.881148 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.881150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.882062 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.882072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.883233 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.883237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.883330 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.883357 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 07:00:11.883405 (+ 48us) client_negotiation.cc:240] Negotiation successful
0812 07:00:11.883476 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":32,"thread_start_us":96,"threads_started":1}
I20260812 07:00:11.884263 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.878210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.27.110.194:44113)
0812 07:00:11.879319 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.879332 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.879354 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:00:11.879425 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.879431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.879628 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.879791 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:11.879949 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.881012 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.882201 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.883034 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.883813 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.883941 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.883997 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.884083 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":154,"threads_started":1}
I20260812 07:00:11.884951 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.877436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:36345 (local address 127.0.0.1:51958)
0812 07:00:11.878381 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:11.878396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:11.878486 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:11.879481 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:11.879484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:11.879499 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:11.879603 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.879609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.881091 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.881094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.883251 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:11.883259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.884581 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.884585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:11.884675 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.884697 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:00:11.884768 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 07:00:11.884829 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260812 07:00:11.885344 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:11.878125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51958 (local address 127.27.110.193:36345)
0812 07:00:11.878833 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:11.878849 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:11.878868 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:00:11.878931 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:11.878935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:11.879137 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:11.879299 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:11.879717 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.880925 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.883384 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:11.884208 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:11.884340 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:11.884476 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:11.885126 (+ 650us) server_negotiation.cc:299] Negotiation successful
0812 07:00:11.885233 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":91,"threads_started":1}
I20260812 07:00:11.886965 29716 tablet_service.cc:1467] Processing CreateTablet for tablet 1068ee60a6cc40468fa50d24c00ca5be (DEFAULT_TABLE table=test-table [id=079b4bdbc8194446a75a148095124552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:11.886965 29990 tablet_service.cc:1467] Processing CreateTablet for tablet 1068ee60a6cc40468fa50d24c00ca5be (DEFAULT_TABLE table=test-table [id=079b4bdbc8194446a75a148095124552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:11.888255 29716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1068ee60a6cc40468fa50d24c00ca5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:11.887861 29452 tablet_service.cc:1467] Processing CreateTablet for tablet 1068ee60a6cc40468fa50d24c00ca5be (DEFAULT_TABLE table=test-table [id=079b4bdbc8194446a75a148095124552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:11.888922 29452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1068ee60a6cc40468fa50d24c00ca5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:11.890278 29990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1068ee60a6cc40468fa50d24c00ca5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:11.895390 30157 tablet_bootstrap.cc:492] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0: Bootstrap starting.
I20260812 07:00:11.897100 30160 tablet_bootstrap.cc:492] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: Bootstrap starting.
I20260812 07:00:11.897578 30157 tablet_bootstrap.cc:654] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:11.897859 30167 tablet_bootstrap.cc:492] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: Bootstrap starting.
I20260812 07:00:11.899596 30157 log.cc:824] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:11.900053 30160 tablet_bootstrap.cc:654] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:11.900525 30167 tablet_bootstrap.cc:654] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:11.900969 30160 log.cc:824] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:11.901332 30167 log.cc:824] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:11.901875 30157 tablet_bootstrap.cc:492] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0: No bootstrap required, opened a new log
I20260812 07:00:11.902108 30157 ts_tablet_manager.cc:1397] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:00:11.902946 30160 tablet_bootstrap.cc:492] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: No bootstrap required, opened a new log
I20260812 07:00:11.903162 30160 ts_tablet_manager.cc:1397] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:00:11.905715 30157 raft_consensus.cc:348] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:11.905988 30157 raft_consensus.cc:374] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:11.906085 30157 raft_consensus.cc:729] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f58d12cab9048d591f8d7c97522faf0, State: Initialized, Role: FOLLOWER
I20260812 07:00:11.906528 30157 consensus_queue.cc:260] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:11.906728 30160 raft_consensus.cc:348] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:11.906999 30160 raft_consensus.cc:374] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:11.907059 30160 raft_consensus.cc:729] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76981da8dbde4bf09beb05fd5ace8103, State: Initialized, Role: FOLLOWER
I20260812 07:00:11.907337 29567 heartbeater.cc:503] Master 127.27.110.254:39041 was elected leader, sending a full tablet report...
I20260812 07:00:11.907629 30157 ts_tablet_manager.cc:1428] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:11.907518 30160 consensus_queue.cc:260] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:11.908651 29836 heartbeater.cc:503] Master 127.27.110.254:39041 was elected leader, sending a full tablet report...
I20260812 07:00:11.908763 30160 ts_tablet_manager.cc:1428] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260812 07:00:11.909677 30167 tablet_bootstrap.cc:492] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: No bootstrap required, opened a new log
I20260812 07:00:11.909884 30167 ts_tablet_manager.cc:1397] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260812 07:00:11.913968 30167 raft_consensus.cc:348] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:11.914309 30167 raft_consensus.cc:374] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:11.914389 30167 raft_consensus.cc:729] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea2d0c15d9ea43819de6ca7876259613, State: Initialized, Role: FOLLOWER
I20260812 07:00:11.914834 30167 consensus_queue.cc:260] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:11.915488 30128 heartbeater.cc:503] Master 127.27.110.254:39041 was elected leader, sending a full tablet report...
I20260812 07:00:11.915752 30167 ts_tablet_manager.cc:1428] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260812 07:00:11.999120 30178 raft_consensus.cc:482] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:11.999320 30178 raft_consensus.cc:504] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:12.000494 30178 leader_election.cc:288] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea2d0c15d9ea43819de6ca7876259613 (127.27.110.195:36753), 7f58d12cab9048d591f8d7c97522faf0 (127.27.110.193:36345)
I20260812 07:00:12.004884 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.000913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:50161 (local address 127.27.110.195:36753)
0812 07:00:12.001081 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:12.001104 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:12.001124 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:00:12.001182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:12.001187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:12.001254 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:12.001339 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:12.001725 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.002375 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.003588 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.004213 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.004309 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.004349 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:12.004628 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 07:00:12.004737 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:00:12.004914 29842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.000680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:36753 (local address 127.27.110.194:50161)
0812 07:00:12.000834 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:12.000853 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:12.000946 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:12.001365 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:12.001367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:12.001388 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:12.001587 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:12.001595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.002484 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.002488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:12.003455 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:12.003465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.004332 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.004336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:12.004457 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.004485 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:00:12.004583 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 07:00:12.004728 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":29,"spinlock_wait_cycles":3328}
I20260812 07:00:12.005053 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.000963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:33053 (local address 127.27.110.193:36345)
0812 07:00:12.001113 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:12.001118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:12.001436 (+ 318us) server_negotiation.cc:407] Connection header received
0812 07:00:12.001578 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:12.001581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:12.001684 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:12.001765 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:12.002052 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.002647 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.003787 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.004374 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.004551 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.004611 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:12.004849 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 07:00:12.004924 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:00:12.005164 30182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.000868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:36345 (local address 127.27.110.194:33053)
0812 07:00:12.001323 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:12.001339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:12.001465 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:12.001806 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:12.001809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:12.001828 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:12.001941 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:12.001947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.002765 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.002768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:12.003606 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:12.003617 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.004499 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.004502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:12.004658 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.004683 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 07:00:12.004808 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 07:00:12.004938 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":1,"spinlock_wait_cycles":19072,"thread_start_us":171,"threads_started":1}
I20260812 07:00:12.005977 29484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1068ee60a6cc40468fa50d24c00ca5be" candidate_uuid: "76981da8dbde4bf09beb05fd5ace8103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f58d12cab9048d591f8d7c97522faf0" is_pre_election: true
I20260812 07:00:12.006400 29484 raft_consensus.cc:2454] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76981da8dbde4bf09beb05fd5ace8103 in term 0.
I20260812 07:00:12.006973 29618 leader_election.cc:302] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76981da8dbde4bf09beb05fd5ace8103, 7f58d12cab9048d591f8d7c97522faf0; no voters:
I20260812 07:00:12.007303 30178 raft_consensus.cc:2781] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:12.007402 30178 raft_consensus.cc:482] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:12.007458 30178 raft_consensus.cc:3037] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:12.008096 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1068ee60a6cc40468fa50d24c00ca5be" candidate_uuid: "76981da8dbde4bf09beb05fd5ace8103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea2d0c15d9ea43819de6ca7876259613" is_pre_election: true
I20260812 07:00:12.008422 30026 raft_consensus.cc:2454] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76981da8dbde4bf09beb05fd5ace8103 in term 0.
I20260812 07:00:12.008723 30178 raft_consensus.cc:504] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:12.009114 30178 leader_election.cc:288] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [CANDIDATE]: Term 1 election: Requested vote from peers ea2d0c15d9ea43819de6ca7876259613 (127.27.110.195:36753), 7f58d12cab9048d591f8d7c97522faf0 (127.27.110.193:36345)
I20260812 07:00:12.009473 30026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1068ee60a6cc40468fa50d24c00ca5be" candidate_uuid: "76981da8dbde4bf09beb05fd5ace8103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea2d0c15d9ea43819de6ca7876259613"
I20260812 07:00:12.009481 29484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1068ee60a6cc40468fa50d24c00ca5be" candidate_uuid: "76981da8dbde4bf09beb05fd5ace8103" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f58d12cab9048d591f8d7c97522faf0"
I20260812 07:00:12.009589 29484 raft_consensus.cc:3037] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:12.009598 30026 raft_consensus.cc:3037] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:12.010675 29484 raft_consensus.cc:2454] T 1068ee60a6cc40468fa50d24c00ca5be P 7f58d12cab9048d591f8d7c97522faf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76981da8dbde4bf09beb05fd5ace8103 in term 1.
I20260812 07:00:12.010731 30026 raft_consensus.cc:2454] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76981da8dbde4bf09beb05fd5ace8103 in term 1.
I20260812 07:00:12.011003 29618 leader_election.cc:302] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76981da8dbde4bf09beb05fd5ace8103, 7f58d12cab9048d591f8d7c97522faf0; no voters:
I20260812 07:00:12.011209 30178 raft_consensus.cc:2781] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:12.011480 30178 raft_consensus.cc:686] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [term 1 LEADER]: Becoming Leader. State: Replica: 76981da8dbde4bf09beb05fd5ace8103, State: Running, Role: LEADER
I20260812 07:00:12.011831 30178 consensus_queue.cc:237] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } }
I20260812 07:00:12.015215 29192 catalog_manager.cc:5495] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76981da8dbde4bf09beb05fd5ace8103 (127.27.110.194). New cstate: current_term: 1 leader_uuid: "76981da8dbde4bf09beb05fd5ace8103" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea2d0c15d9ea43819de6ca7876259613" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 36753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f58d12cab9048d591f8d7c97522faf0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 36345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76981da8dbde4bf09beb05fd5ace8103" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44113 } health_report { overall_health: HEALTHY } } }
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:12.041697 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.027859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.27.110.193:36345)
0812 07:00:12.028015 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:12.028019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:12.028188 (+ 169us) server_negotiation.cc:407] Connection header received
0812 07:00:12.028340 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:12.028348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:12.028388 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:12.028476 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:12.029173 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.029681 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.030552 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.030905 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.031019 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.037915 (+ 6896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:12.037930 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:12.037953 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:12.037980 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:12.039799 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:12.040254 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:12.040261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:12.040263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:12.040354 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:12.040674 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:12.040677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:12.040679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:12.040999 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:12.041229 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:12.041384 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 07:00:12.041512 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:12.071413 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.062432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.27.110.193:36345)
0812 07:00:12.062591 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:12.062598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:12.062731 (+ 133us) server_negotiation.cc:407] Connection header received
0812 07:00:12.062850 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:12.062852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:12.062897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:12.062969 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:12.063757 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.064299 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.065157 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.065390 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.065473 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.068070 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:12.068074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:12.068076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:12.068102 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:12.069605 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:12.069996 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:12.069999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:12.070000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:12.070086 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:12.070581 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:12.070584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:12.070585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:12.070746 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:12.070845 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:12.071179 (+ 334us) server_negotiation.cc:299] Negotiation successful
0812 07:00:12.071286 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260812 07:00:12.080463 30131 tablet.cc:2369] T 1068ee60a6cc40468fa50d24c00ca5be P ea2d0c15d9ea43819de6ca7876259613: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[29099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[29099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:12.099963 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:12.091169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51984 (local address 127.27.110.193:36345)
0812 07:00:12.091328 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:12.091333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:12.091515 (+ 182us) server_negotiation.cc:407] Connection header received
0812 07:00:12.091680 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:12.091683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:12.091727 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:12.091794 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:12.092693 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.093197 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.094263 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:12.094470 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:12.094547 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:12.096994 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:12.096997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:12.096999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:12.097022 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:12.098296 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:12.098681 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:12.098683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:12.098685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:12.098756 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:12.098993 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:12.098995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:12.098996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:12.099212 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:12.099314 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:12.099607 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 07:00:12.099808 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 07:00:12.101666 29414 tablet_service.cc:2908] Rejecting scan request for tablet 1068ee60a6cc40468fa50d24c00ca5be: Uninitialized: clean time has not yet been initialized
I20260812 07:00:12.104138 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29315
W20260812 07:00:12.129128 29839 tablet.cc:2369] T 1068ee60a6cc40468fa50d24c00ca5be P 76981da8dbde4bf09beb05fd5ace8103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:12.138342 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29584
I20260812 07:00:12.152220 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29849
I20260812 07:00:12.163769 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29118
2026-08-12T07:00:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2701 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[30211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[30211](info): set up 2 sockets
Aug 12 07:00:12 dist-test-slave-w4v5 krb5kdc[30211](info): commencing operation
krb5kdc: starting...
W20260812 07:00:13.473811 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:13 dist-test-slave-w4v5 krb5kdc[30211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:13.599777 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:35965
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:35965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:13.711611 30242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:13.711930 30242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:13.712014 30242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:13.716058 30242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:13.716144 30242 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 07:00:13.716178 30242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:13.716248 30242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:13.716288 30242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:13.716325 30242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:13.721601 30242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:35965
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:35965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:13.722730 30242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:13.723845 30242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:13.732712 30253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:13.732712 30250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:13.732787 30249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:13.733284 30242 server_base.cc:1034] running on GCE node
I20260812 07:00:13.733906 30242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:13.735013 30242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:13.736263 30242 hybrid_clock.cc:648] HybridClock initialized: now 1786518013736240 us; error 32 us; skew 500 ppm
Aug 12 07:00:13 dist-test-slave-w4v5 krb5kdc[30211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:13.739220 30242 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:13.740120 30242 webserver.cc:460] Webserver started at http://127.27.110.254:44917/ using document root <none> and password file <none>
I20260812 07:00:13.740684 30242 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:13.740777 30242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:13.741024 30242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:13.743220 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "9e8aa845b27e498daebe2d6905850ab7"
format_stamp: "Formatted at 2026-08-12 07:00:13 on dist-test-slave-w4v5"
server_key: "4b84a36cc2f9f4f7853fa764fbbd5ea7"
server_key_iv: "5072736bd8bbb2a90f6cd153b703208b"
server_key_version: "encryptionkey@0"
I20260812 07:00:13.743811 30242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "9e8aa845b27e498daebe2d6905850ab7"
format_stamp: "Formatted at 2026-08-12 07:00:13 on dist-test-slave-w4v5"
server_key: "4b84a36cc2f9f4f7853fa764fbbd5ea7"
server_key_iv: "5072736bd8bbb2a90f6cd153b703208b"
server_key_version: "encryptionkey@0"
I20260812 07:00:13.747691 30242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:00:13.750193 30271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:13.751513 30242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:13.751655 30242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "9e8aa845b27e498daebe2d6905850ab7"
format_stamp: "Formatted at 2026-08-12 07:00:13 on dist-test-slave-w4v5"
server_key: "4b84a36cc2f9f4f7853fa764fbbd5ea7"
server_key_iv: "5072736bd8bbb2a90f6cd153b703208b"
server_key_version: "encryptionkey@0"
I20260812 07:00:13.751766 30242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:13.773662 30242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:13.776778 30242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:13.776993 30242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:13.785637 30242 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:35965
I20260812 07:00:13.785665 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:35965 every 8 connection(s)
I20260812 07:00:13.786648 30242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:13.789422 30399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:13.795058 30399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: Bootstrap starting.
I20260812 07:00:13.796213 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30242
I20260812 07:00:13.796311 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:13.796578 28091 external_mini_cluster.cc:1428] Setting key 61ae8946e8d3deddaf158d4ed197748d
I20260812 07:00:13.797904 30399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:13.798698 30399 log.cc:824] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:13.800711 30399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: No bootstrap required, opened a new log
I20260812 07:00:13.804976 30399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8aa845b27e498daebe2d6905850ab7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35965 } }
I20260812 07:00:13.805310 30399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:13.805365 30399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8aa845b27e498daebe2d6905850ab7, State: Initialized, Role: FOLLOWER
Aug 12 07:00:13 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:13.805824 30399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8aa845b27e498daebe2d6905850ab7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35965 } }
I20260812 07:00:13.805987 30399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:13.806069 30399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:13.806211 30399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:13.807482 30399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8aa845b27e498daebe2d6905850ab7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35965 } }
I20260812 07:00:13.807844 30399 leader_election.cc:302] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e8aa845b27e498daebe2d6905850ab7; no voters:
I20260812 07:00:13.808235 30399 leader_election.cc:288] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:13.808316 30412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:13.808531 30412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [term 1 LEADER]: Becoming Leader. State: Replica: 9e8aa845b27e498daebe2d6905850ab7, State: Running, Role: LEADER
I20260812 07:00:13.808871 30412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8aa845b27e498daebe2d6905850ab7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35965 } }
I20260812 07:00:13.809703 30399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:13.810945 30417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e8aa845b27e498daebe2d6905850ab7. Latest consensus state: current_term: 1 leader_uuid: "9e8aa845b27e498daebe2d6905850ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8aa845b27e498daebe2d6905850ab7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35965 } } }
I20260812 07:00:13.811208 30417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:13.811602 30416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e8aa845b27e498daebe2d6905850ab7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e8aa845b27e498daebe2d6905850ab7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35965 } } }
I20260812 07:00:13.811791 30416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:13.812004 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:13.797717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.27.110.254:35965)
0812 07:00:13.798198 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:13.798214 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:13.798249 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:00:13.798937 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:13.798957 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:13.799397 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:13.799786 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:13.800585 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:13.801629 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:13.802550 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:13.803022 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:13.803248 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:13.806132 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:13.806140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:13.806158 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:13.806192 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:13.808192 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:13.808831 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:13.808839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:13.808847 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:13.809032 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:13.809402 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:13.809406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:13.809407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:13.809945 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:13.810136 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:13.810641 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 07:00:13.810966 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":145,"threads_started":1}
W20260812 07:00:13.815239 30443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:13.815308 30443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:13.815387 30429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:13.816344 30429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:13.822463 30429 catalog_manager.cc:1347] Generated new cluster ID: 6ec4e88f05ed4be790e3d9f937ae500c
I20260812 07:00:13.822566 30429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:13.855377 30429 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:13.856561 30429 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:13.864516 30429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: Generated new TSK 0
I20260812 07:00:13.865247 30429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:13.901684 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:35965
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:14.020287 30454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:14.020543 30454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:14.020622 30454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:14.024555 30454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:14.024608 30454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:14.024683 30454 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:14.024766 30454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:14.030021 30454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:14.031061 30454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:14.032161 30454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:14.042017 30468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:14.041997 30465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:14.042186 30454 server_base.cc:1034] running on GCE node
W20260812 07:00:14.042430 30464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:14.043012 30454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:14.043874 30454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:14.045063 30454 hybrid_clock.cc:648] HybridClock initialized: now 1786518014045042 us; error 73 us; skew 500 ppm
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:14.047892 30454 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:14.048871 30454 webserver.cc:460] Webserver started at http://127.27.110.193:40143/ using document root <none> and password file <none>
I20260812 07:00:14.049424 30454 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:14.049484 30454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:14.049674 30454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:14.051769 30454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "a1fe6917f01f42558e1392795def75d7"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "3b7af06c853de7719d9c506f216afcb1"
server_key_iv: "ff73bbe55e17a4fcb3432c8b640980c9"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.052371 30454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "a1fe6917f01f42558e1392795def75d7"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "3b7af06c853de7719d9c506f216afcb1"
server_key_iv: "ff73bbe55e17a4fcb3432c8b640980c9"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.056327 30454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 07:00:14.058790 30485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:14.060083 30454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:14.060221 30454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "a1fe6917f01f42558e1392795def75d7"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "3b7af06c853de7719d9c506f216afcb1"
server_key_iv: "ff73bbe55e17a4fcb3432c8b640980c9"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.060329 30454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:14.072613 30454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:14.075462 30454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:14.075656 30454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:14.077080 30454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:14.077160 30454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:14.077263 30454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:14.077291 30454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:14.088339 30454 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:43677
I20260812 07:00:14.088459 30694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:43677 every 8 connection(s)
I20260812 07:00:14.089516 30454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:14.091666 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.077785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:34089 (local address 127.27.110.254:35965)
0812 07:00:14.077918 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.077923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.078965 (+ 1042us) server_negotiation.cc:407] Connection header received
0812 07:00:14.079949 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.079953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.080013 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.080123 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.081839 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.082525 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.083857 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.084141 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.084257 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.087497 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.087500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.087502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.087527 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.089315 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.089903 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.089909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.089912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.090018 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.090472 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.090475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.090477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.090671 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.090783 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.091341 (+ 558us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.091483 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 07:00:14.092363 30503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.078287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.27.110.193:34089)
0812 07:00:14.078813 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.078856 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.079739 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.080411 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.080424 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.080941 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:14.081645 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.081671 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.082663 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.082667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.083722 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.083733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.084304 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.084308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.084531 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.085122 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:14.085144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:14.087320 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:14.089431 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.089435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.089446 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.089778 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.090147 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.090153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.090155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.090333 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.090782 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:14.090789 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:14.091033 (+ 244us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.091281 (+ 248us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.091591 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":142,"threads_started":1}
I20260812 07:00:14.093678 30695 heartbeater.cc:347] Connected to a master server at 127.27.110.254:35965
I20260812 07:00:14.093814 30695 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:14.094298 30695 heartbeater.cc:511] Master 127.27.110.254:35965 requested a full tablet report, sending...
I20260812 07:00:14.095780 30308 ts_manager.cc:194] Registered new tserver with Master: a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677)
I20260812 07:00:14.097270 30308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:34089
I20260812 07:00:14.098976 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30454
I20260812 07:00:14.099126 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:14.099449 28091 external_mini_cluster.cc:1428] Setting key 1150da46af17cd5bb7b67a450b40d69b
I20260812 07:00:14.103657 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.096105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:45907 (local address 127.27.110.254:35965)
0812 07:00:14.096232 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.096237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.096396 (+ 159us) server_negotiation.cc:407] Connection header received
0812 07:00:14.096500 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.096502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.096537 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.096599 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.097064 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.097613 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.098557 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.098860 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.098967 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.100327 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.100334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.100337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.100371 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.101962 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.102395 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.102400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.102401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.102488 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.102785 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.102789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.102790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.102963 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.103074 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.103375 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.103487 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:14.103667 30704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.096042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.27.110.193:45907)
0812 07:00:14.096309 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.096324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.096419 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.096644 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.096647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.096853 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:14.096961 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.096967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.097752 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.097755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.098433 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.098442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.098993 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.099000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.099130 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.099672 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:14.099687 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:14.100138 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:14.102084 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.102089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.102091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.102292 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.102590 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.102593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.102595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.102685 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.103064 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:14.103067 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:14.103198 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.103308 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.103458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:14.128376 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:35965
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:14.239934 30708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:14.240177 30708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:14.240224 30708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:14.244161 30708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:14.244210 30708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:14.244247 30708 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:14.244279 30708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:14.249369 30708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:14.250414 30708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:14.251480 30708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:14.261579 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:14.261579 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:14.261633 30716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:14.262101 30708 server_base.cc:1034] running on GCE node
I20260812 07:00:14.262526 30708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:14.263281 30708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:14.264483 30708 hybrid_clock.cc:648] HybridClock initialized: now 1786518014264460 us; error 34 us; skew 500 ppm
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:14.267329 30708 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:14.268241 30708 webserver.cc:460] Webserver started at http://127.27.110.194:36475/ using document root <none> and password file <none>
I20260812 07:00:14.268811 30708 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:14.268904 30708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:14.269119 30708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:14.271262 30708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "2e72ac3006db4399aabf34cabd99cc00"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "f5c2fe93e9328ed46b4d66870969ddc0"
server_key_iv: "54814a7fe2912a9449134616ca77ed0d"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.271843 30708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "2e72ac3006db4399aabf34cabd99cc00"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "f5c2fe93e9328ed46b4d66870969ddc0"
server_key_iv: "54814a7fe2912a9449134616ca77ed0d"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.275645 30708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:00:14.278151 30738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:14.279381 30708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:14.279517 30708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "2e72ac3006db4399aabf34cabd99cc00"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "f5c2fe93e9328ed46b4d66870969ddc0"
server_key_iv: "54814a7fe2912a9449134616ca77ed0d"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.279636 30708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:14.297561 30708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:14.300689 30708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:14.300899 30708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:14.302347 30708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:14.302423 30708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:14.302498 30708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:14.302542 30708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:14.314299 30708 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:36379
I20260812 07:00:14.314337 30950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:36379 every 8 connection(s)
I20260812 07:00:14.315317 30708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:14.317444 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.303412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57575 (local address 127.27.110.254:35965)
0812 07:00:14.303621 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.303624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.304597 (+ 973us) server_negotiation.cc:407] Connection header received
0812 07:00:14.305756 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.305762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.305832 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.305935 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.307474 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.308014 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.309194 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.309423 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.309539 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.313587 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.313590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.313592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.313618 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.315381 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.315956 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.315960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.315962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.316058 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.316449 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.316489 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.316490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.316678 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.316764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.317213 (+ 449us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.317318 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 07:00:14.318186 30759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.303919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.27.110.194:57575)
0812 07:00:14.304416 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.304497 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.305517 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.306149 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.306160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.306645 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:14.307311 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.307327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.308147 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.308158 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.309047 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.309057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.309591 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.309595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.309785 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.310362 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:14.310390 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:14.313411 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:14.315500 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.315505 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.315518 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.315834 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.316149 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.316152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.316154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.316329 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.316774 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:14.316779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:14.316988 (+ 209us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.317146 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.317374 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":150,"threads_started":1}
I20260812 07:00:14.319396 30951 heartbeater.cc:347] Connected to a master server at 127.27.110.254:35965
I20260812 07:00:14.319528 30951 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:14.320066 30951 heartbeater.cc:511] Master 127.27.110.254:35965 requested a full tablet report, sending...
I20260812 07:00:14.321177 30310 ts_manager.cc:194] Registered new tserver with Master: 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194:36379)
I20260812 07:00:14.321903 30310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:57575
I20260812 07:00:14.325004 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30708
I20260812 07:00:14.325119 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:14.325398 28091 external_mini_cluster.cc:1428] Setting key dfe8d4b9c318a4fe41674cad2343f7ea
I20260812 07:00:14.328961 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.321721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:34035 (local address 127.27.110.254:35965)
0812 07:00:14.321850 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.321856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.322030 (+ 174us) server_negotiation.cc:407] Connection header received
0812 07:00:14.322265 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.322268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.322322 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.322398 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.322867 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.323628 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.324428 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.324648 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.324729 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.325792 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.325795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.325797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.325817 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.327314 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.327783 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.327786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.327788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.327869 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.328142 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.328145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.328148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.328352 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.328410 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.328707 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.328810 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:00:14.328982 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.321668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.27.110.194:34035)
0812 07:00:14.321932 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.321948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.322047 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.322432 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.322435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.322646 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:14.322751 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.322756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.323769 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.323772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.324326 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.324334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.324762 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.324765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.324830 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.325287 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:14.325299 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:14.325655 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:14.327437 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.327440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.327442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.327697 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.327951 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.327954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.327956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.328040 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.328473 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:14.328475 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:14.328543 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.328654 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.328793 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:14.354146 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:35965
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:14.468705 30967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:14.468941 30967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:14.468984 30967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:14.473093 30967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:14.473148 30967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:14.473186 30967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:14.473253 30967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:14.478327 30967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:35965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:14.479367 30967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:14.480432 30967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:00:14.490180 30967 server_base.cc:1034] running on GCE node
W20260812 07:00:14.490209 30977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:14.490240 30976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:14.490427 30979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:14.490847 30967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:14.491508 30967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:14.492671 30967 hybrid_clock.cc:648] HybridClock initialized: now 1786518014492656 us; error 27 us; skew 500 ppm
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:14.495741 30967 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:14.496600 30967 webserver.cc:460] Webserver started at http://127.27.110.195:34043/ using document root <none> and password file <none>
I20260812 07:00:14.497161 30967 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:14.497269 30967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:14.497506 30967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:14.499621 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "05822df1db908f3dcc187fe4cefca4f0"
server_key_iv: "88e5461d1dc362af44405f439e2417f9"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.500208 30967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "05822df1db908f3dcc187fe4cefca4f0"
server_key_iv: "88e5461d1dc362af44405f439e2417f9"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.503940 30967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:14.506394 30992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:14.507754 30967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:14.507896 30967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595"
format_stamp: "Formatted at 2026-08-12 07:00:14 on dist-test-slave-w4v5"
server_key: "05822df1db908f3dcc187fe4cefca4f0"
server_key_iv: "88e5461d1dc362af44405f439e2417f9"
server_key_version: "encryptionkey@0"
I20260812 07:00:14.508008 30967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:14.519757 30967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:14.523067 30967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:14.523355 30967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:14.524849 30967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:14.524925 30967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:14.524986 30967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:14.525040 30967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:14.536691 30967 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:38251
I20260812 07:00:14.536811 31214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:38251 every 8 connection(s)
I20260812 07:00:14.537861 30967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:14.539960 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.525611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:53185 (local address 127.27.110.254:35965)
0812 07:00:14.525796 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.525803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.527297 (+ 1494us) server_negotiation.cc:407] Connection header received
0812 07:00:14.528528 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.528533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.528602 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.528707 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.530083 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.530692 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.532618 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.532926 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.533055 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.535815 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.535819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.535821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.535851 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.537700 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.538324 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.538327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.538330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.538432 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.538877 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.538884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.538886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.539067 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.539223 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.539739 (+ 516us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.539841 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:00:14.540840 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.526401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.27.110.195:53185)
0812 07:00:14.527121 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.527173 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.528266 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.528882 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.528889 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.529277 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:14.529924 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.529940 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.530841 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.530846 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.532474 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.532490 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.533109 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.533113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.533305 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.533923 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:14.533939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:14.535644 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:14.537836 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.537841 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.537852 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.538204 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.538606 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.538610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.538613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.538771 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.539274 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:14.539280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:14.539497 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.539666 (+ 169us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.539904 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":360,"threads_started":1}
I20260812 07:00:14.541224 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30967
I20260812 07:00:14.541347 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:14.541656 28091 external_mini_cluster.cc:1428] Setting key 2fa807dbf1baa517e63255cee4d68eda
I20260812 07:00:14.542423 31215 heartbeater.cc:347] Connected to a master server at 127.27.110.254:35965
I20260812 07:00:14.542578 31215 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:14.543754 31215 heartbeater.cc:511] Master 127.27.110.254:35965 requested a full tablet report, sending...
I20260812 07:00:14.544829 30308 ts_manager.cc:194] Registered new tserver with Master: cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251)
I20260812 07:00:14.545614 30308 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:53185
I20260812 07:00:14.551448 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.544691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:40525 (local address 127.27.110.254:35965)
0812 07:00:14.544809 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.544813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.544972 (+ 159us) server_negotiation.cc:407] Connection header received
0812 07:00:14.545135 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.545138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.545175 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.545231 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.545728 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.546222 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.547033 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.547314 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.547423 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.548407 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.548411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.548415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.548441 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.549726 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.550184 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.550187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.550189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.550272 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.550592 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.550595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.550597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.550775 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.550886 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.551183 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.551292 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 07:00:14.551457 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.544565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.27.110.195:40525)
0812 07:00:14.544886 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.544904 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.545033 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.545273 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.545276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.545479 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:14.545592 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.545599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.546351 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.546354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.546911 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.546919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.547431 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.547434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.547532 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.547946 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:14.547957 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:14.548263 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:14.549866 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.549869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.549871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.550073 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.550369 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:14.550373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:14.550375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:14.550485 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:14.550878 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:14.550881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:14.550982 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.551109 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.551270 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
I20260812 07:00:14.555708 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:00:14 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:14.583206 30406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.574757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.27.110.254:35965)
0812 07:00:14.574897 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.574901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.574914 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:00:14.574952 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.574955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.574994 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.575106 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:14.575847 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.576376 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.577236 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.577456 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.577564 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.580005 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:14.580009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:14.580014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:14.580044 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:14.581424 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.581870 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.581875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.581877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.581964 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:14.582289 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:14.582293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:14.582294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:14.582483 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:14.582633 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.582893 (+ 260us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.583012 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 07:00:14.586078 30308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:14.589210 30308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:14.605156 31246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.598271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:43677 (local address 127.0.0.1:51186)
0812 07:00:14.599361 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.599374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.599445 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.600221 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.600227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.600265 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:14.600408 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.600415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.601852 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.601855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.603416 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.603425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.604640 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.604646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.604747 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.604810 (+ 63us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.604874 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.604956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1037,"mutex_wait_us":25,"spinlock_wait_cycles":5376,"thread_start_us":113,"threads_started":1}
I20260812 07:00:14.605772 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.598827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.27.110.194:36379)
0812 07:00:14.599653 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.599661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.599680 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:00:14.599741 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.599745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.599923 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.600080 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:14.600560 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.601273 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.602727 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.603529 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.603695 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.603859 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.605438 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.605575 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":88,"threads_started":1}
I20260812 07:00:14.605787 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.598887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.27.110.193:43677)
0812 07:00:14.599661 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.599668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.599690 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:00:14.599762 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.599766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.599933 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.600053 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:14.600549 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.601739 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.603554 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.604326 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.604450 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.604580 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.605514 (+ 934us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.605628 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":87,"threads_started":1}
I20260812 07:00:14.605985 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.598021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36379 (local address 127.0.0.1:40876)
0812 07:00:14.599196 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.599211 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.599341 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.600223 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.600226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.600261 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:14.600398 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.600404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.601403 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.601406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.602594 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.602604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.603931 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.603938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.604075 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.604133 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.604245 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.604384 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"mutex_wait_us":51,"spinlock_wait_cycles":8320,"thread_start_us":91,"threads_started":1}
I20260812 07:00:14.606359 31248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.597990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38251 (local address 127.0.0.1:39262)
0812 07:00:14.598656 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.598711 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.599346 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.599963 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.599969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.599999 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:14.600140 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.600148 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.601782 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.601789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.602932 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.602942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.606003 (+ 3061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.606007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.606101 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.606125 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.606175 (+ 50us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.606240 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":18,"thread_start_us":80,"threads_started":1}
I20260812 07:00:14.607115 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.598698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.27.110.195:38251)
0812 07:00:14.599195 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.599210 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.599242 (+ 32us) server_negotiation.cc:407] Connection header received
0812 07:00:14.599540 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.599547 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.599780 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.599942 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:14.600272 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.601636 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.603057 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.605856 (+ 2799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.606670 (+ 814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.606817 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.606858 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.606929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":115,"threads_started":1}
I20260812 07:00:14.608809 30567 tablet_service.cc:1467] Processing CreateTablet for tablet e74b240df9a44393b91147502949f826 (DEFAULT_TABLE table=test-table [id=1c51910ead4940a09e2462a4a7a2962b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:14.609166 30829 tablet_service.cc:1467] Processing CreateTablet for tablet e74b240df9a44393b91147502949f826 (DEFAULT_TABLE table=test-table [id=1c51910ead4940a09e2462a4a7a2962b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:14.609599 31096 tablet_service.cc:1467] Processing CreateTablet for tablet e74b240df9a44393b91147502949f826 (DEFAULT_TABLE table=test-table [id=1c51910ead4940a09e2462a4a7a2962b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:14.610093 30567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74b240df9a44393b91147502949f826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:14.610253 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74b240df9a44393b91147502949f826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:14.610620 31096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e74b240df9a44393b91147502949f826. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:14.616478 31255 tablet_bootstrap.cc:492] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Bootstrap starting.
I20260812 07:00:14.618381 31259 tablet_bootstrap.cc:492] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Bootstrap starting.
I20260812 07:00:14.619469 31260 tablet_bootstrap.cc:492] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Bootstrap starting.
I20260812 07:00:14.619760 31255 tablet_bootstrap.cc:654] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:14.620605 31259 tablet_bootstrap.cc:654] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:14.620687 31255 log.cc:824] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:14.621372 31259 log.cc:824] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:14.621505 31260 tablet_bootstrap.cc:654] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:14.622363 31260 log.cc:824] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:14.622807 31255 tablet_bootstrap.cc:492] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: No bootstrap required, opened a new log
I20260812 07:00:14.623018 31255 ts_tablet_manager.cc:1397] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 07:00:14.623456 31259 tablet_bootstrap.cc:492] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: No bootstrap required, opened a new log
I20260812 07:00:14.623667 31259 ts_tablet_manager.cc:1397] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:14.624155 31260 tablet_bootstrap.cc:492] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: No bootstrap required, opened a new log
I20260812 07:00:14.624341 31260 ts_tablet_manager.cc:1397] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 07:00:14.627501 31260 raft_consensus.cc:348] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.627461 31259 raft_consensus.cc:348] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.627741 31259 raft_consensus.cc:374] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:14.627740 31260 raft_consensus.cc:374] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:14.627800 31260 raft_consensus.cc:729] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e72ac3006db4399aabf34cabd99cc00, State: Initialized, Role: FOLLOWER
I20260812 07:00:14.627800 31259 raft_consensus.cc:729] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0fcbc6d23c4fb79ae2a114e9e6a595, State: Initialized, Role: FOLLOWER
I20260812 07:00:14.627655 31255 raft_consensus.cc:348] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.627869 31255 raft_consensus.cc:374] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:14.627926 31255 raft_consensus.cc:729] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fe6917f01f42558e1392795def75d7, State: Initialized, Role: FOLLOWER
I20260812 07:00:14.628268 31260 consensus_queue.cc:260] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.628273 31259 consensus_queue.cc:260] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.628325 31255 consensus_queue.cc:260] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.628935 30695 heartbeater.cc:503] Master 127.27.110.254:35965 was elected leader, sending a full tablet report...
I20260812 07:00:14.629248 31259 ts_tablet_manager.cc:1428] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:00:14.629248 31255 ts_tablet_manager.cc:1428] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260812 07:00:14.629256 31260 ts_tablet_manager.cc:1428] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:14.629559 31215 heartbeater.cc:503] Master 127.27.110.254:35965 was elected leader, sending a full tablet report...
I20260812 07:00:14.630025 30951 heartbeater.cc:503] Master 127.27.110.254:35965 was elected leader, sending a full tablet report...
I20260812 07:00:14.646503 31272 raft_consensus.cc:482] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:14.646690 31272 raft_consensus.cc:504] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.647727 31272 leader_election.cc:288] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194:36379), a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677)
I20260812 07:00:14.651944 31252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.648041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:34331 (local address 127.27.110.194:36379)
0812 07:00:14.648156 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.648161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.648482 (+ 321us) server_negotiation.cc:407] Connection header received
0812 07:00:14.648586 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.648589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.648633 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.648725 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:14.649039 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.649583 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.650660 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.651299 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.651422 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.651500 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.651783 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.651854 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 07:00:14.651971 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.647965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36379 (local address 127.27.110.195:34331)
0812 07:00:14.648352 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.648368 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.648472 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.648741 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.648743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.648758 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:14.648901 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.648907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.649719 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.649722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.650528 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.650537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.651442 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.651445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.651605 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.651638 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.651735 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.651804 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":49,"thread_start_us":129,"threads_started":1}
I20260812 07:00:14.652570 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74b240df9a44393b91147502949f826" candidate_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e72ac3006db4399aabf34cabd99cc00" is_pre_election: true
I20260812 07:00:14.652774 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.648041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:49733 (local address 127.27.110.193:43677)
0812 07:00:14.648156 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.648161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.648266 (+ 105us) server_negotiation.cc:407] Connection header received
0812 07:00:14.648400 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.648406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.648460 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.648549 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:14.648869 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.649460 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.651536 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.652206 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.652317 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.652363 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.652597 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.652666 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:00:14.652791 31011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.647965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:43677 (local address 127.27.110.195:49733)
0812 07:00:14.648174 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:14.648191 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:14.648288 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:14.648587 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:14.648590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:14.648610 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:14.648751 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.648759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.649640 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.649646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.651402 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:14.651413 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.652315 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.652318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:14.652430 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.652447 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 07:00:14.652583 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 07:00:14.652660 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 07:00:14.652860 30866 raft_consensus.cc:2454] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0fcbc6d23c4fb79ae2a114e9e6a595 in term 0.
I20260812 07:00:14.653385 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74b240df9a44393b91147502949f826" candidate_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fe6917f01f42558e1392795def75d7" is_pre_election: true
I20260812 07:00:14.653393 31001 leader_election.cc:302] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e72ac3006db4399aabf34cabd99cc00, cc0fcbc6d23c4fb79ae2a114e9e6a595; no voters:
I20260812 07:00:14.653625 30618 raft_consensus.cc:2454] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0fcbc6d23c4fb79ae2a114e9e6a595 in term 0.
I20260812 07:00:14.653641 31272 raft_consensus.cc:2781] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:14.653725 31272 raft_consensus.cc:482] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:14.653779 31272 raft_consensus.cc:3037] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:14.654980 31272 raft_consensus.cc:504] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.655385 31272 leader_election.cc:288] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [CANDIDATE]: Term 1 election: Requested vote from peers 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194:36379), a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677)
I20260812 07:00:14.655681 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74b240df9a44393b91147502949f826" candidate_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e72ac3006db4399aabf34cabd99cc00"
I20260812 07:00:14.655803 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e74b240df9a44393b91147502949f826" candidate_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fe6917f01f42558e1392795def75d7"
I20260812 07:00:14.655812 30866 raft_consensus.cc:3037] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:14.655920 30618 raft_consensus.cc:3037] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:14.656984 30618 raft_consensus.cc:2454] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0fcbc6d23c4fb79ae2a114e9e6a595 in term 1.
I20260812 07:00:14.657313 30998 leader_election.cc:302] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1fe6917f01f42558e1392795def75d7, cc0fcbc6d23c4fb79ae2a114e9e6a595; no voters:
I20260812 07:00:14.657501 31272 raft_consensus.cc:2781] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:14.657691 30866 raft_consensus.cc:2454] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0fcbc6d23c4fb79ae2a114e9e6a595 in term 1.
I20260812 07:00:14.657797 31272 raft_consensus.cc:686] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 LEADER]: Becoming Leader. State: Replica: cc0fcbc6d23c4fb79ae2a114e9e6a595, State: Running, Role: LEADER
I20260812 07:00:14.658149 31272 consensus_queue.cc:237] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } }
I20260812 07:00:14.661324 30307 catalog_manager.cc:5495] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195). New cstate: current_term: 1 leader_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:14.688198 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:14.684485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.27.110.195:38251)
0812 07:00:14.684608 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:14.684611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:14.684627 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:14.684827 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:14.684829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:14.684881 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:14.684965 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:14.685222 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.685717 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.686841 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:14.687049 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:14.687199 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:14.687462 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:14.687874 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:14.687961 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:14.688046 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 07:00:14.688095 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 07:00:14.694489 30866 raft_consensus.cc:1261] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Refusing update from remote peer cc0fcbc6d23c4fb79ae2a114e9e6a595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:14.695375 31274 consensus_queue.cc:1035] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:14.695779 30618 raft_consensus.cc:1261] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Refusing update from remote peer cc0fcbc6d23c4fb79ae2a114e9e6a595: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:14.696326 31274 consensus_queue.cc:1035] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:14.705343 31283 mvcc.cc:205] Tried to move back new op lower bound from 7317577788179968000 to 7317577788042477568. Current Snapshot: MvccSnapshot[applied={T|T < 7317577788179968000}]
I20260812 07:00:14.707753 31288 mvcc.cc:205] Tried to move back new op lower bound from 7317577788179968000 to 7317577788042477568. Current Snapshot: MvccSnapshot[applied={T|T < 7317577788179968000}]
I20260812 07:00:14.715849 30307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:14.716060 30307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:14.718710 30307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: Sending DeleteTablet for 3 replicas of tablet e74b240df9a44393b91147502949f826
I20260812 07:00:14.719555 31096 tablet_service.cc:1514] Processing DeleteTablet for tablet e74b240df9a44393b91147502949f826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:14 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:39262
I20260812 07:00:14.719664 30829 tablet_service.cc:1514] Processing DeleteTablet for tablet e74b240df9a44393b91147502949f826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:14 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:40876
I20260812 07:00:14.720175 31301 tablet_replica.cc:330] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: stopping tablet replica
I20260812 07:00:14.720191 30567 tablet_service.cc:1514] Processing DeleteTablet for tablet e74b240df9a44393b91147502949f826 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:14 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:51186
I20260812 07:00:14.720515 31301 raft_consensus.cc:2229] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:14.720705 31306 tablet_replica.cc:330] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: stopping tablet replica
I20260812 07:00:14.720907 31301 raft_consensus.cc:2258] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:14.721104 31306 raft_consensus.cc:2229] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:14.721455 31306 raft_consensus.cc:2258] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:14.721379 30307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:14.721808 30307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:14.722534 31301 ts_tablet_manager.cc:1905] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:14.723049 31306 ts_tablet_manager.cc:1905] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:14.725625 31301 ts_tablet_manager.cc:1918] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:14.725729 31301 log.cc:1196] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/e74b240df9a44393b91147502949f826
I20260812 07:00:14.726095 31301 ts_tablet_manager.cc:1939] T e74b240df9a44393b91147502949f826 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Deleting consensus metadata
I20260812 07:00:14.726666 31306 ts_tablet_manager.cc:1918] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:14.726831 31306 log.cc:1196] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/e74b240df9a44393b91147502949f826
I20260812 07:00:14.727239 31306 ts_tablet_manager.cc:1939] T e74b240df9a44393b91147502949f826 P a1fe6917f01f42558e1392795def75d7: Deleting consensus metadata
I20260812 07:00:14.727523 30272 catalog_manager.cc:4841] TS cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251): tablet e74b240df9a44393b91147502949f826 (table test-table [id=1c51910ead4940a09e2462a4a7a2962b]) successfully deleted
I20260812 07:00:14.727603 30829 tablet_service.cc:1467] Processing CreateTablet for tablet 120076f408a54921b522183039080b71 (DEFAULT_TABLE table=test-table [id=1f06ec9972a44d50a3debd4bf948ff99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:14.727907 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120076f408a54921b522183039080b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:14.728154 30567 tablet_service.cc:1467] Processing CreateTablet for tablet 120076f408a54921b522183039080b71 (DEFAULT_TABLE table=test-table [id=1f06ec9972a44d50a3debd4bf948ff99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:14.728415 30567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120076f408a54921b522183039080b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:14.728788 30276 catalog_manager.cc:4841] TS a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677): tablet e74b240df9a44393b91147502949f826 (table test-table [id=1c51910ead4940a09e2462a4a7a2962b]) successfully deleted
I20260812 07:00:14.730551 31260 tablet_bootstrap.cc:492] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Bootstrap starting.
I20260812 07:00:14.731127 31096 tablet_service.cc:1467] Processing CreateTablet for tablet 120076f408a54921b522183039080b71 (DEFAULT_TABLE table=test-table [id=1f06ec9972a44d50a3debd4bf948ff99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:14.731379 31096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120076f408a54921b522183039080b71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:14.731565 31260 tablet_bootstrap.cc:654] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:14.731904 31304 tablet_replica.cc:330] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: stopping tablet replica
I20260812 07:00:14.732199 31255 tablet_bootstrap.cc:492] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Bootstrap starting.
I20260812 07:00:14.732195 31304 raft_consensus.cc:2229] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:14.732506 31304 raft_consensus.cc:2258] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:14.733098 31255 tablet_bootstrap.cc:654] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:14.733685 31259 tablet_bootstrap.cc:492] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Bootstrap starting.
I20260812 07:00:14.734614 31259 tablet_bootstrap.cc:654] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:14.734822 31255 tablet_bootstrap.cc:492] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: No bootstrap required, opened a new log
I20260812 07:00:14.734892 31255 ts_tablet_manager.cc:1397] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 07:00:14.735004 31260 tablet_bootstrap.cc:492] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: No bootstrap required, opened a new log
I20260812 07:00:14.735100 31260 ts_tablet_manager.cc:1397] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 07:00:14.735548 31260 raft_consensus.cc:348] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:14.735564 31255 raft_consensus.cc:348] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:14.735656 31260 raft_consensus.cc:374] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:14.735683 31255 raft_consensus.cc:374] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:14.735718 31255 raft_consensus.cc:729] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fe6917f01f42558e1392795def75d7, State: Initialized, Role: FOLLOWER
I20260812 07:00:14.735687 31260 raft_consensus.cc:729] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e72ac3006db4399aabf34cabd99cc00, State: Initialized, Role: FOLLOWER
I20260812 07:00:14.735848 31255 consensus_queue.cc:260] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:14.736004 31260 consensus_queue.cc:260] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:14.736099 31255 ts_tablet_manager.cc:1428] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:14.736249 31260 ts_tablet_manager.cc:1428] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:14.736481 31304 ts_tablet_manager.cc:1905] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:14.739657 31304 ts_tablet_manager.cc:1918] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:14.739739 31304 log.cc:1196] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/e74b240df9a44393b91147502949f826
I20260812 07:00:14.740029 31304 ts_tablet_manager.cc:1939] T e74b240df9a44393b91147502949f826 P 2e72ac3006db4399aabf34cabd99cc00: Deleting consensus metadata
I20260812 07:00:14.741226 30277 catalog_manager.cc:4841] TS 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194:36379): tablet e74b240df9a44393b91147502949f826 (table test-table [id=1c51910ead4940a09e2462a4a7a2962b]) successfully deleted
I20260812 07:00:14.741889 31259 tablet_bootstrap.cc:492] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: No bootstrap required, opened a new log
I20260812 07:00:14.741984 31259 ts_tablet_manager.cc:1397] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260812 07:00:14.742429 31259 raft_consensus.cc:348] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:14.742573 31259 raft_consensus.cc:374] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:14.742619 31259 raft_consensus.cc:729] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0fcbc6d23c4fb79ae2a114e9e6a595, State: Initialized, Role: FOLLOWER
I20260812 07:00:14.742736 31259 consensus_queue.cc:260] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:14.742990 31259 ts_tablet_manager.cc:1428] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260812 07:00:14.789021 31217 tablet.cc:2369] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:00:14.816572 30952 tablet.cc:2369] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:00:14.840943 30696 tablet.cc:2369] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:15.024631 31270 raft_consensus.cc:482] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:15.025053 31270 raft_consensus.cc:504] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.026571 31270 leader_election.cc:288] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677), cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251)
I20260812 07:00:15.030740 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.026972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:46091 (local address 127.27.110.195:38251)
0812 07:00:15.027121 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.027126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.027355 (+ 229us) server_negotiation.cc:407] Connection header received
0812 07:00:15.027487 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.027490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.027536 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.027626 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:15.027936 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.028430 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.029690 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.030116 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.030252 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.030314 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.030527 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.030602 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 07:00:15.030766 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.026845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38251 (local address 127.27.110.194:46091)
0812 07:00:15.027225 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.027237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.027375 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.027658 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.027661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.027703 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:15.027803 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.027808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.028661 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.028665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.029538 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.029548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.030247 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.030250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.030351 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.030374 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.030507 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.030581 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":50,"spinlock_wait_cycles":768,"thread_start_us":83,"threads_started":1}
I20260812 07:00:15.030794 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.026882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:60919 (local address 127.27.110.193:43677)
0812 07:00:15.027022 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.027027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.027305 (+ 278us) server_negotiation.cc:407] Connection header received
0812 07:00:15.027452 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.027456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.027518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.027587 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:15.027919 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.028429 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.029728 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.030191 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.030320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.030371 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.030608 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.030688 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:00:15.030810 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.026744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:43677 (local address 127.27.110.194:60919)
0812 07:00:15.027191 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.027212 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.027333 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.027628 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.027631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.027648 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:15.027798 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.027804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.028605 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.028645 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.029544 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.029555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.030334 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.030338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.030471 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.030489 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.030582 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.030650 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":10,"spinlock_wait_cycles":14720,"thread_start_us":93,"threads_started":1}
I20260812 07:00:15.031733 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120076f408a54921b522183039080b71" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fe6917f01f42558e1392795def75d7" is_pre_election: true
I20260812 07:00:15.031695 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120076f408a54921b522183039080b71" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" is_pre_election: true
I20260812 07:00:15.031936 30618 raft_consensus.cc:2454] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 0.
I20260812 07:00:15.032027 31133 raft_consensus.cc:2454] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 0.
I20260812 07:00:15.032347 30746 leader_election.cc:302] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e72ac3006db4399aabf34cabd99cc00, a1fe6917f01f42558e1392795def75d7; no voters:
I20260812 07:00:15.032645 31270 raft_consensus.cc:2781] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:15.032709 31270 raft_consensus.cc:482] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:15.032748 31270 raft_consensus.cc:3037] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:15.033854 31270 raft_consensus.cc:504] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.034200 31270 leader_election.cc:288] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 election: Requested vote from peers a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677), cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251)
I20260812 07:00:15.034541 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120076f408a54921b522183039080b71" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fe6917f01f42558e1392795def75d7"
I20260812 07:00:15.034606 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120076f408a54921b522183039080b71" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595"
I20260812 07:00:15.034682 30618 raft_consensus.cc:3037] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:15.034761 31133 raft_consensus.cc:3037] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:15.035729 30618 raft_consensus.cc:2454] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 1.
I20260812 07:00:15.035897 31133 raft_consensus.cc:2454] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 1.
I20260812 07:00:15.036226 30739 leader_election.cc:302] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e72ac3006db4399aabf34cabd99cc00, cc0fcbc6d23c4fb79ae2a114e9e6a595; no voters:
I20260812 07:00:15.036410 31270 raft_consensus.cc:2781] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:15.036620 31270 raft_consensus.cc:686] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 LEADER]: Becoming Leader. State: Replica: 2e72ac3006db4399aabf34cabd99cc00, State: Running, Role: LEADER
I20260812 07:00:15.036898 31270 consensus_queue.cc:237] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.039197 30308 catalog_manager.cc:5495] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194). New cstate: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:00:15 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518013, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:15.137200 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.128844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:35965 (local address 127.0.0.1:41178)
0812 07:00:15.129137 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.129152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.129271 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.129529 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.129532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.129992 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:15.130092 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.130098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.130870 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.130873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.131505 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.131515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.131947 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.131950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.132012 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.132533 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:15.132544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:15.134142 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:15.135786 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:15.135795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:15.135799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:15.136031 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:15.136278 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:15.136280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:15.136281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:15.136361 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:15.136716 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:15.136722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:15.136823 (+ 101us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.136913 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.137037 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
I20260812 07:00:15.137228 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.128891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.27.110.254:35965)
0812 07:00:15.129176 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.129180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.129230 (+ 50us) server_negotiation.cc:407] Connection header received
0812 07:00:15.129356 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.129358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.129400 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.129487 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:15.130202 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.130775 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.131623 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.131849 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.131924 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.134267 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:15.134270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:15.134272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:15.134293 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:15.135685 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:15.136117 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:15.136120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:15.136121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:15.136198 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:15.136437 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:15.136439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:15.136440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:15.136620 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:15.136696 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.136971 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.137097 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":214,"thread_start_us":128,"threads_started":1}
I20260812 07:00:15.141301 30308 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:41178:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 07:00:15.142206 30308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:15.147332 30829 tablet_service.cc:1467] Processing CreateTablet for tablet 86b12b21b06740a5914a05715bcd9fea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6df5292c122e453481ca956aed183e35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:15.147574 31096 tablet_service.cc:1467] Processing CreateTablet for tablet 86b12b21b06740a5914a05715bcd9fea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6df5292c122e453481ca956aed183e35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:15.147805 30829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b12b21b06740a5914a05715bcd9fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:15.148006 31096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b12b21b06740a5914a05715bcd9fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:15.147332 30567 tablet_service.cc:1467] Processing CreateTablet for tablet 86b12b21b06740a5914a05715bcd9fea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6df5292c122e453481ca956aed183e35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:15.149029 30567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b12b21b06740a5914a05715bcd9fea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:15.150630 31259 tablet_bootstrap.cc:492] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: Bootstrap starting.
I20260812 07:00:15.150820 31260 tablet_bootstrap.cc:492] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: Bootstrap starting.
I20260812 07:00:15.151635 31259 tablet_bootstrap.cc:654] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:15.151856 31260 tablet_bootstrap.cc:654] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:15.152201 31255 tablet_bootstrap.cc:492] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: Bootstrap starting.
I20260812 07:00:15.153107 31259 tablet_bootstrap.cc:492] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: No bootstrap required, opened a new log
I20260812 07:00:15.153127 31255 tablet_bootstrap.cc:654] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:15.153223 31259 ts_tablet_manager.cc:1397] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 07:00:15.153715 31259 raft_consensus.cc:348] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.153805 31260 tablet_bootstrap.cc:492] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: No bootstrap required, opened a new log
I20260812 07:00:15.153842 31259 raft_consensus.cc:374] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:15.153882 31260 ts_tablet_manager.cc:1397] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:15.153877 31259 raft_consensus.cc:729] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0fcbc6d23c4fb79ae2a114e9e6a595, State: Initialized, Role: FOLLOWER
I20260812 07:00:15.154021 31259 consensus_queue.cc:260] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.154392 31255 tablet_bootstrap.cc:492] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: No bootstrap required, opened a new log
I20260812 07:00:15.154336 31274 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.154376 31260 raft_consensus.cc:348] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.154469 31274 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.154469 31260 raft_consensus.cc:374] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:15.154476 31255 ts_tablet_manager.cc:1397] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:00:15.154505 31260 raft_consensus.cc:729] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e72ac3006db4399aabf34cabd99cc00, State: Initialized, Role: FOLLOWER
I20260812 07:00:15.154634 31260 consensus_queue.cc:260] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.154721 31259 ts_tablet_manager.cc:1428] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:15.154901 31260 ts_tablet_manager.cc:1428] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:15.155000 31255 raft_consensus.cc:348] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.155161 31255 raft_consensus.cc:374] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:15.155128 31270 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.155205 31255 raft_consensus.cc:729] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1fe6917f01f42558e1392795def75d7, State: Initialized, Role: FOLLOWER
I20260812 07:00:15.155243 31270 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.155344 31255 consensus_queue.cc:260] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.155630 31255 ts_tablet_manager.cc:1428] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:15.155961 31267 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.156127 31267 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.357263 31267 raft_consensus.cc:482] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:15.357461 31267 raft_consensus.cc:504] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.357656 31270 raft_consensus.cc:482] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:15.357780 31270 raft_consensus.cc:504] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.358289 31270 leader_election.cc:288] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677), cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251)
I20260812 07:00:15.358747 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b12b21b06740a5914a05715bcd9fea" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fe6917f01f42558e1392795def75d7" is_pre_election: true
I20260812 07:00:15.358785 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b12b21b06740a5914a05715bcd9fea" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" is_pre_election: true
I20260812 07:00:15.358880 30618 raft_consensus.cc:2454] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 0.
I20260812 07:00:15.358898 31267 leader_election.cc:288] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194:36379), cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251)
I20260812 07:00:15.358979 31133 raft_consensus.cc:2454] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 0.
I20260812 07:00:15.359546 30739 leader_election.cc:302] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2e72ac3006db4399aabf34cabd99cc00, a1fe6917f01f42558e1392795def75d7, cc0fcbc6d23c4fb79ae2a114e9e6a595; no voters:
I20260812 07:00:15.359768 31270 raft_consensus.cc:2781] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:15.359830 31270 raft_consensus.cc:482] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:15.359872 31270 raft_consensus.cc:3037] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:15.360882 31270 raft_consensus.cc:504] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.361266 31270 leader_election.cc:288] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 election: Requested vote from peers a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677), cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251)
I20260812 07:00:15.361593 30618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b12b21b06740a5914a05715bcd9fea" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1fe6917f01f42558e1392795def75d7"
I20260812 07:00:15.361655 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b12b21b06740a5914a05715bcd9fea" candidate_uuid: "2e72ac3006db4399aabf34cabd99cc00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595"
I20260812 07:00:15.361702 30618 raft_consensus.cc:3037] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:15.361765 31133 raft_consensus.cc:3037] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:15.362835 30618 raft_consensus.cc:2454] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 1.
I20260812 07:00:15.362938 31133 raft_consensus.cc:2454] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e72ac3006db4399aabf34cabd99cc00 in term 1.
I20260812 07:00:15.363325 30739 leader_election.cc:302] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e72ac3006db4399aabf34cabd99cc00, cc0fcbc6d23c4fb79ae2a114e9e6a595; no voters:
I20260812 07:00:15.363556 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.359125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36379 (local address 127.27.110.193:55523)
0812 07:00:15.359495 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.359512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.359606 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.360140 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.360143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.360162 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:15.360288 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.360293 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.361071 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.361084 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.362126 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.362136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.362812 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.362816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.362917 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.362947 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.363036 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.363354 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":83,"threads_started":1}
I20260812 07:00:15.363592 31270 raft_consensus.cc:2781] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:15.363680 31270 raft_consensus.cc:686] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [term 1 LEADER]: Becoming Leader. State: Replica: 2e72ac3006db4399aabf34cabd99cc00, State: Running, Role: LEADER
I20260812 07:00:15.363821 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.359343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:55523 (local address 127.27.110.194:36379)
0812 07:00:15.359897 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.359903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.359918 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:00:15.359964 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.359967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.360011 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.360119 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:15.360400 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.360955 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.362254 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.362687 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.362819 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.362876 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.363166 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.363254 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":65,"threads_started":1}
I20260812 07:00:15.363853 31270 consensus_queue.cc:237] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } }
I20260812 07:00:15.364509 31313 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: TxnStatusTablet state changed. Reason: New leader 2e72ac3006db4399aabf34cabd99cc00. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.364627 31313 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:15.364861 30866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b12b21b06740a5914a05715bcd9fea" candidate_uuid: "a1fe6917f01f42558e1392795def75d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e72ac3006db4399aabf34cabd99cc00" is_pre_election: true
I20260812 07:00:15.364948 31342 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 07:00:15.365160 31342 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 07:00:15.366328 30308 catalog_manager.cc:5495] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194). New cstate: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:15.367291 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.359227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38251 (local address 127.27.110.193:39803)
0812 07:00:15.361474 (+ 2247us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.361491 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.363169 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.363581 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.363586 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.363606 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:15.363745 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.363751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.364722 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.364726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.365837 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.365852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.366804 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.366808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.366910 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.366935 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.367064 (+ 129us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.367155 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2179,"thread_start_us":88,"threads_started":1}
I20260812 07:00:15.367293 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.359702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:39803 (local address 127.27.110.195:38251)
0812 07:00:15.359855 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.359862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.361598 (+ 1736us) server_negotiation.cc:407] Connection header received
0812 07:00:15.363317 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.363320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.363370 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.363421 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:15.363882 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.364553 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.365975 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.366671 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.366791 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.366839 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.367106 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.367190 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:00:15.368016 31133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86b12b21b06740a5914a05715bcd9fea" candidate_uuid: "a1fe6917f01f42558e1392795def75d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" is_pre_election: true
I20260812 07:00:15.368230 31133 raft_consensus.cc:2379] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a1fe6917f01f42558e1392795def75d7 in current term 1: Already voted for candidate 2e72ac3006db4399aabf34cabd99cc00 in this term.
I20260812 07:00:15.368707 30486 leader_election.cc:302] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a1fe6917f01f42558e1392795def75d7; no voters: 2e72ac3006db4399aabf34cabd99cc00, cc0fcbc6d23c4fb79ae2a114e9e6a595
I20260812 07:00:15.368974 31267 raft_consensus.cc:2726] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 07:00:15.396068 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.392208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.27.110.194:36379)
0812 07:00:15.392349 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.392354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.392370 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:15.392530 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.392534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.392599 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.392706 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:15.392960 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.393558 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.394509 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.394749 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.394880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.395227 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:15.395665 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:15.395796 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.395869 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.395925 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:00:15.396068 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.392089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36379 (local address 127.0.0.1:48074)
0812 07:00:15.392236 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.392256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.392370 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.392731 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.392734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.392741 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:15.392855 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.392860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.393700 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.393704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.394380 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.394393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.394888 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.394891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.395002 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.395045 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:15.395771 (+ 726us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:15.395788 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.395857 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.395955 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 07:00:15.399540 31133 raft_consensus.cc:1261] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Refusing update from remote peer 2e72ac3006db4399aabf34cabd99cc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:15.399678 30618 raft_consensus.cc:1261] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Refusing update from remote peer 2e72ac3006db4399aabf34cabd99cc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:15.400134 31313 consensus_queue.cc:1035] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:15.400559 31270 consensus_queue.cc:1035] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:15.401780 31267 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: TxnStatusTablet state changed. Reason: New leader 2e72ac3006db4399aabf34cabd99cc00. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.401930 31267 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.402395 31274 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: TxnStatusTablet state changed. Reason: New leader 2e72ac3006db4399aabf34cabd99cc00. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.402536 31274 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.404210 31270 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.404326 31270 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:15.405040 31313 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.405138 31313 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P 2e72ac3006db4399aabf34cabd99cc00: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:15.405355 31274 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.405479 31274 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P cc0fcbc6d23c4fb79ae2a114e9e6a595: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.406078 31267 tablet_replica.cc:439] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2e72ac3006db4399aabf34cabd99cc00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 } } peers { permanent_uuid: "2e72ac3006db4399aabf34cabd99cc00" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36379 } } peers { permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 } } }
I20260812 07:00:15.406203 31267 tablet_replica.cc:442] T 86b12b21b06740a5914a05715bcd9fea P a1fe6917f01f42558e1392795def75d7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:15.406776 31356 mvcc.cc:205] Tried to move back new op lower bound from 7317577791070355456 to 7317577790931828736. Current Snapshot: MvccSnapshot[applied={T|T < 7317577791070355456}]
I20260812 07:00:15.416896 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.413174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.27.110.194:36379)
0812 07:00:15.413293 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.413297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.413404 (+ 107us) server_negotiation.cc:407] Connection header received
0812 07:00:15.413548 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.413550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.413599 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.413664 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:15.413955 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.414442 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.415676 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.415893 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.415973 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.416335 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:15.416401 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:15.416458 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.416643 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.416695 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:00:15.428067 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.424421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:58879 (local address 127.27.110.194:36379)
0812 07:00:15.424572 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.424576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.424723 (+ 147us) server_negotiation.cc:407] Connection header received
0812 07:00:15.424838 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.424841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.424880 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.424942 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:15.425222 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.425690 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.426924 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.427207 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.427287 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.427649 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:15.427714 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:15.427775 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.427900 (+ 125us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.427940 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260812 07:00:15.428182 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.424294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36379 (local address 127.27.110.194:58879)
0812 07:00:15.424617 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:15.424633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:15.424746 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:15.424984 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:15.424987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:15.424995 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:15.425120 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.425126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.425795 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.425798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.426800 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:15.426811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.427317 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.427321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:15.427420 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.427473 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:15.427807 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:15.427814 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:00:15.427882 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 07:00:15.427967 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260812 07:00:15.435249 30618 raft_consensus.cc:1261] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Refusing update from remote peer 2e72ac3006db4399aabf34cabd99cc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:15.435614 31133 raft_consensus.cc:1261] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Refusing update from remote peer 2e72ac3006db4399aabf34cabd99cc00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:15.435654 31270 consensus_queue.cc:1035] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1fe6917f01f42558e1392795def75d7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:15.436045 31270 consensus_queue.cc:1035] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc0fcbc6d23c4fb79ae2a114e9e6a595" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:15.476825 30308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:15.476981 30308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:15.479104 30308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e8aa845b27e498daebe2d6905850ab7: Sending DeleteTablet for 3 replicas of tablet 120076f408a54921b522183039080b71
I20260812 07:00:15.479701 30567 tablet_service.cc:1514] Processing DeleteTablet for tablet 120076f408a54921b522183039080b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:51186
I20260812 07:00:15.479723 30829 tablet_service.cc:1514] Processing DeleteTablet for tablet 120076f408a54921b522183039080b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:40876
I20260812 07:00:15.479878 31096 tablet_service.cc:1514] Processing DeleteTablet for tablet 120076f408a54921b522183039080b71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:39262
I20260812 07:00:15.480134 31378 tablet_replica.cc:330] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: stopping tablet replica
I20260812 07:00:15.480265 31378 raft_consensus.cc:2229] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:15.480473 31379 tablet_replica.cc:330] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: stopping tablet replica
I20260812 07:00:15.480584 31378 raft_consensus.cc:2258] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:15.480957 31379 raft_consensus.cc:2229] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:15.481141 31379 raft_consensus.cc:2258] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:15.481247 31378 ts_tablet_manager.cc:1905] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:15.481402 31376 tablet_replica.cc:330] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: stopping tablet replica
I20260812 07:00:15.481534 31376 raft_consensus.cc:2229] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:15.481680 31376 raft_consensus.cc:2258] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:15.482053 31379 ts_tablet_manager.cc:1905] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:15.482312 31376 ts_tablet_manager.cc:1905] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:15.485097 31378 ts_tablet_manager.cc:1918] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:15.485201 31378 log.cc:1196] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/120076f408a54921b522183039080b71
I20260812 07:00:15.485373 31376 ts_tablet_manager.cc:1918] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:15.485481 31376 log.cc:1196] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/120076f408a54921b522183039080b71
I20260812 07:00:15.485508 31378 ts_tablet_manager.cc:1939] T 120076f408a54921b522183039080b71 P 2e72ac3006db4399aabf34cabd99cc00: Deleting consensus metadata
I20260812 07:00:15.485786 31376 ts_tablet_manager.cc:1939] T 120076f408a54921b522183039080b71 P a1fe6917f01f42558e1392795def75d7: Deleting consensus metadata
I20260812 07:00:15.486012 31379 ts_tablet_manager.cc:1918] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:15.486150 31379 log.cc:1196] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/120076f408a54921b522183039080b71
I20260812 07:00:15.486318 30277 catalog_manager.cc:4841] TS 2e72ac3006db4399aabf34cabd99cc00 (127.27.110.194:36379): tablet 120076f408a54921b522183039080b71 (table test-table [id=1f06ec9972a44d50a3debd4bf948ff99]) successfully deleted
I20260812 07:00:15.486461 30276 catalog_manager.cc:4841] TS a1fe6917f01f42558e1392795def75d7 (127.27.110.193:43677): tablet 120076f408a54921b522183039080b71 (table test-table [id=1f06ec9972a44d50a3debd4bf948ff99]) successfully deleted
I20260812 07:00:15.486629 31379 ts_tablet_manager.cc:1939] T 120076f408a54921b522183039080b71 P cc0fcbc6d23c4fb79ae2a114e9e6a595: Deleting consensus metadata
I20260812 07:00:15.487322 30272 catalog_manager.cc:4841] TS cc0fcbc6d23c4fb79ae2a114e9e6a595 (127.27.110.195:38251): tablet 120076f408a54921b522183039080b71 (table test-table [id=1f06ec9972a44d50a3debd4bf948ff99]) successfully deleted
Aug 12 07:00:15 dist-test-slave-w4v5 krb5kdc[30211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518014, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:15.498461 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.487527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41332 (local address 127.27.110.193:43677)
0812 07:00:15.487685 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.487692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.488040 (+ 348us) server_negotiation.cc:407] Connection header received
0812 07:00:15.488208 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.488211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.488262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.488351 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:15.489057 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.489573 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.490422 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.490659 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.490775 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.494235 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:15.494246 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:15.494259 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:15.494285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:15.496664 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:15.497155 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:15.497160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:15.497162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:15.497239 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:15.497536 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:15.497539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:15.497540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:15.497777 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:15.497992 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.498191 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.498295 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 07:00:15.499032 30688 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41332
I20260812 07:00:15.509117 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:15.500945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.27.110.254:35965)
0812 07:00:15.501079 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:15.501084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:15.501295 (+ 211us) server_negotiation.cc:407] Connection header received
0812 07:00:15.501410 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:15.501412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:15.501453 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:15.501517 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:15.502349 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.502858 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.504020 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:15.504256 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:15.504336 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:15.505915 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:15.505918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:15.505920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:15.505944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:15.507394 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:15.507881 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:15.507883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:15.507885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:15.507958 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:15.508259 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:15.508261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:15.508262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:15.508432 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:15.508534 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:15.508822 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 07:00:15.508930 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260812 07:00:15.509548 30308 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44776
I20260812 07:00:15.511010 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30454
I20260812 07:00:15.526499 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30708
I20260812 07:00:15.546358 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30967
I20260812 07:00:15.567008 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30242
2026-08-12T07:00:15Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3405 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:15 dist-test-slave-w4v5 krb5kdc[31395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:15 dist-test-slave-w4v5 krb5kdc[31395](info): set up 2 sockets
Aug 12 07:00:15 dist-test-slave-w4v5 krb5kdc[31395](info): commencing operation
krb5kdc: starting...
W20260812 07:00:16.870074 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:16 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:17.006937 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39245
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:17.124341 31433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:17.124584 31433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:17.124622 31433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:17.128575 31433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:17.128630 31433 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 07:00:17.128647 31433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:17.128666 31433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:17.128682 31433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:17.128746 31433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:17.133940 31433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39245
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:17.134986 31433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:17.136031 31433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:17.144573 31444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.144544 31441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.144573 31448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:17.145057 31433 server_base.cc:1034] running on GCE node
I20260812 07:00:17.145625 31433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:17.147121 31433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:17.148314 31433 hybrid_clock.cc:648] HybridClock initialized: now 1786518017148298 us; error 35 us; skew 500 ppm
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:17.154471 31433 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:17.155568 31433 webserver.cc:460] Webserver started at http://127.27.110.254:34481/ using document root <none> and password file <none>
I20260812 07:00:17.156167 31433 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:17.156261 31433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:17.156502 31433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:17.158641 31433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "9b003ec6655e45bb821388253ec55142"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "c1dd303137cf4f29202f57954e784fb0"
server_key_iv: "ceba8f237e4b73dca6181cc94d285bdf"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.159277 31433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "9b003ec6655e45bb821388253ec55142"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "c1dd303137cf4f29202f57954e784fb0"
server_key_iv: "ceba8f237e4b73dca6181cc94d285bdf"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.163198 31433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:00:17.165658 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.166956 31433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:17.167162 31433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "9b003ec6655e45bb821388253ec55142"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "c1dd303137cf4f29202f57954e784fb0"
server_key_iv: "ceba8f237e4b73dca6181cc94d285bdf"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.167284 31433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:17.178960 31433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:17.182325 31433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:17.182529 31433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:17.191179 31433 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39245
I20260812 07:00:17.191191 31600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39245 every 8 connection(s)
I20260812 07:00:17.192190 31433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:17.194594 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 31433
I20260812 07:00:17.194691 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:17.194985 28091 external_mini_cluster.cc:1428] Setting key ebf71a1b1de565030a057dbf6452659a
I20260812 07:00:17.194993 31602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:17.202302 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Bootstrap starting.
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:17.204906 31602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:17.205770 31602 log.cc:824] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:17.208271 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: No bootstrap required, opened a new log
I20260812 07:00:17.211419 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.196473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.27.110.254:39245)
0812 07:00:17.196903 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.196922 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.196963 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:00:17.197834 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.197858 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.198301 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.198681 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.199459 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.200997 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.201877 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.202572 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.202728 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.205431 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.205437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.205452 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.205482 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.208195 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.208675 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.208681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.208687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.208829 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.209169 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.209172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.209174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.209668 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.209867 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.210136 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.210429 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260812 07:00:17.212244 31602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b003ec6655e45bb821388253ec55142" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39245 } }
I20260812 07:00:17.212503 31602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:17.212558 31602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b003ec6655e45bb821388253ec55142, State: Initialized, Role: FOLLOWER
I20260812 07:00:17.213016 31602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b003ec6655e45bb821388253ec55142" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39245 } }
I20260812 07:00:17.213138 31602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:17.213219 31602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:17.213316 31602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:17.214298 31602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b003ec6655e45bb821388253ec55142" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39245 } }
I20260812 07:00:17.214627 31602 leader_election.cc:302] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b003ec6655e45bb821388253ec55142; no voters:
I20260812 07:00:17.214926 31602 leader_election.cc:288] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:17.215119 31608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:17.215394 31608 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [term 1 LEADER]: Becoming Leader. State: Replica: 9b003ec6655e45bb821388253ec55142, State: Running, Role: LEADER
I20260812 07:00:17.215701 31608 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b003ec6655e45bb821388253ec55142" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39245 } }
I20260812 07:00:17.216171 31602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:17.217413 31609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b003ec6655e45bb821388253ec55142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b003ec6655e45bb821388253ec55142" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39245 } } }
I20260812 07:00:17.217551 31609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:17.217749 31610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b003ec6655e45bb821388253ec55142. Latest consensus state: current_term: 1 leader_uuid: "9b003ec6655e45bb821388253ec55142" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b003ec6655e45bb821388253ec55142" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39245 } } }
I20260812 07:00:17.217904 31610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:17.218314 31627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:17.221048 31627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:17.226408 31627 catalog_manager.cc:1347] Generated new cluster ID: 7ff8bb861bb04e3ca02bbd23c0bd5f7b
I20260812 07:00:17.226472 31627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:17.241108 31627 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:17.242110 31627 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:17.246192 31627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Generated new TSK 0
I20260812 07:00:17.246871 31627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:17.282454 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39245
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:17.395193 31654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:17.395473 31654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:17.395556 31654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:17.399648 31654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:17.399719 31654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:17.399793 31654 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 07:00:17.399834 31654 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:17.399900 31654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:17.404959 31654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:17.405989 31654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:17.406976 31654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:17.416723 31665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.416711 31666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.416698 31672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:17.416790 31654 server_base.cc:1034] running on GCE node
I20260812 07:00:17.417346 31654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:17.418081 31654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:17.419260 31654 hybrid_clock.cc:648] HybridClock initialized: now 1786518017419242 us; error 29 us; skew 500 ppm
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:17.422055 31654 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:17.422855 31654 webserver.cc:460] Webserver started at http://127.27.110.193:42287/ using document root <none> and password file <none>
I20260812 07:00:17.423475 31654 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:17.423564 31654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:17.423816 31654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:17.425758 31654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "083ba4042f0e47bab53f2b6f366085c5"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "f2eeb55306d9f636a58869615012a813"
server_key_iv: "c88dbfdc19890c2fbd6b23ecf237dd0f"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.426301 31654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "083ba4042f0e47bab53f2b6f366085c5"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "f2eeb55306d9f636a58869615012a813"
server_key_iv: "c88dbfdc19890c2fbd6b23ecf237dd0f"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.429971 31654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 07:00:17.432337 31693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.433513 31654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:17.433681 31654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "083ba4042f0e47bab53f2b6f366085c5"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "f2eeb55306d9f636a58869615012a813"
server_key_iv: "c88dbfdc19890c2fbd6b23ecf237dd0f"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.433795 31654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:17.472808 31654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:17.476472 31654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:17.476704 31654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:17.478243 31654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:17.478318 31654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.478389 31654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:17.478439 31654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:17.489966 31654 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:33345
I20260812 07:00:17.491005 31654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:17.491591 31937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:33345 every 8 connection(s)
I20260812 07:00:17.493120 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.479241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:39643 (local address 127.27.110.254:39245)
0812 07:00:17.479371 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.479374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.480479 (+ 1105us) server_negotiation.cc:407] Connection header received
0812 07:00:17.481602 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.481605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.481694 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.481762 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.483589 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.484331 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.486229 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.486586 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.486730 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.489463 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.489467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.489470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.489506 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.490956 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.491520 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.491524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.491526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.491618 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.491990 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.491995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.491998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.492203 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.492310 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.492817 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.492944 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 07:00:17.493849 31710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.479817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.27.110.193:39643)
0812 07:00:17.480322 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:17.480365 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:17.481328 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:17.482260 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:17.482271 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:17.482733 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:17.483435 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.483452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.484484 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.484488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.486078 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.486091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.486770 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.486775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.486951 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.487534 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:17.487550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:17.489284 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:17.491115 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.491124 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.491138 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.491405 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.491719 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.491723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.491725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.491893 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.492324 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:17.492332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:17.492573 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 07:00:17.492756 (+ 183us) client_negotiation.cc:240] Negotiation successful
0812 07:00:17.493063 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":160,"threads_started":1}
I20260812 07:00:17.495112 31938 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39245
I20260812 07:00:17.495260 31938 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:17.495800 31938 heartbeater.cc:511] Master 127.27.110.254:39245 requested a full tablet report, sending...
I20260812 07:00:17.497421 31495 ts_manager.cc:194] Registered new tserver with Master: 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345)
I20260812 07:00:17.498524 31495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:39643
I20260812 07:00:17.500113 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 31654
I20260812 07:00:17.500244 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:17.500530 28091 external_mini_cluster.cc:1428] Setting key d8c49f792cf3dc1c8fa2434b7a388239
I20260812 07:00:17.506567 31949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.497291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.27.110.193:32775)
0812 07:00:17.499175 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:17.499187 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:17.499264 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:17.499534 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:17.499536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:17.499758 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:17.499867 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.499875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.500886 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.500889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.501691 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.501704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.502222 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.502225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.502292 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.502794 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:17.502806 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:17.503210 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:17.505027 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.505032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.505037 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.505309 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.505577 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.505580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.505583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.505693 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.506156 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:17.506158 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:17.506234 (+ 76us) client_negotiation.cc:769] Sending connection context
0812 07:00:17.506317 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 07:00:17.506418 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1780,"thread_start_us":108,"threads_started":1}
I20260812 07:00:17.506757 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.497379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:32775 (local address 127.27.110.254:39245)
0812 07:00:17.497488 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.497494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.499257 (+ 1763us) server_negotiation.cc:407] Connection header received
0812 07:00:17.499364 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.499368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.499424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.499491 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.500000 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.500739 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.501814 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.502107 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.502223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.503357 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.503364 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.503370 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.503401 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.504912 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.505410 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.505412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.505413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.505484 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.505806 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.505812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.505816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.506051 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.506164 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.506404 (+ 240us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.506539 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:17.533936 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39245
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:17.657960 31956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:17.658229 31956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:17.658313 31956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:17.662652 31956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:17.662730 31956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:17.662796 31956 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 07:00:17.662837 31956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:17.662902 31956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:17.668071 31956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:17.669082 31956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:17.670092 31956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:17.679839 31965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.679915 31976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:17.679960 31956 server_base.cc:1034] running on GCE node
W20260812 07:00:17.679855 31966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:17.680559 31956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:17.681181 31956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:17.682396 31956 hybrid_clock.cc:648] HybridClock initialized: now 1786518017682358 us; error 49 us; skew 500 ppm
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:17.685346 31956 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:17.686195 31956 webserver.cc:460] Webserver started at http://127.27.110.194:39059/ using document root <none> and password file <none>
I20260812 07:00:17.686757 31956 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:17.686848 31956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:17.687122 31956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:17.689206 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "8491ae8a46ae46f49202ea9834c31206"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "61e27dd09038037fb09a07001729d5ad"
server_key_iv: "975cddd45e26081fce2f7d8c61569389"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.689780 31956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "8491ae8a46ae46f49202ea9834c31206"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "61e27dd09038037fb09a07001729d5ad"
server_key_iv: "975cddd45e26081fce2f7d8c61569389"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.693614 31956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:17.696118 31994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.697345 31956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:17.697476 31956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "8491ae8a46ae46f49202ea9834c31206"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "61e27dd09038037fb09a07001729d5ad"
server_key_iv: "975cddd45e26081fce2f7d8c61569389"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.697590 31956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:17.715346 31956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:17.719028 31956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:17.719290 31956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:17.720837 31956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:17.720916 31956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.720989 31956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:17.721032 31956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:17.732232 31956 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:40359
I20260812 07:00:17.732304 32231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:40359 every 8 connection(s)
I20260812 07:00:17.733155 31956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:17.733600 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 31956
I20260812 07:00:17.733718 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:17.734026 28091 external_mini_cluster.cc:1428] Setting key 4bc857faba1229559ab02d2a3d03ff87
I20260812 07:00:17.736441 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.721725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:48251 (local address 127.27.110.254:39245)
0812 07:00:17.721887 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.721892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.722844 (+ 952us) server_negotiation.cc:407] Connection header received
0812 07:00:17.723997 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.724001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.724059 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.724168 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.725890 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.726837 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.728727 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.729118 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.729250 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.732261 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.732265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.732267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.732298 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.734082 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.734732 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.734736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.734738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.734833 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.735233 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.735237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.735238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.735431 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.735547 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.736054 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.736197 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:00:17.737114 32007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.722216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.27.110.194:48251)
0812 07:00:17.722666 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:17.722716 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:17.723755 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:17.724350 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:17.724360 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:17.724912 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:17.725676 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.725696 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.726954 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.726957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.728558 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.728572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.729276 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.729279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.729420 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.730089 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:17.730114 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:17.732083 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:17.734270 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.734275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.734289 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.734613 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.734929 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.734933 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.734935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.735127 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.735544 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:17.735550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:17.735780 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 07:00:17.735934 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 07:00:17.736179 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":131,"threads_started":1}
I20260812 07:00:17.738286 32232 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39245
I20260812 07:00:17.738512 32232 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:17.739066 32232 heartbeater.cc:511] Master 127.27.110.254:39245 requested a full tablet report, sending...
I20260812 07:00:17.740304 31495 ts_manager.cc:194] Registered new tserver with Master: 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359)
I20260812 07:00:17.741034 31495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:48251
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:00:17.749315 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.740923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:34347 (local address 127.27.110.254:39245)
0812 07:00:17.741019 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.741023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.741506 (+ 483us) server_negotiation.cc:407] Connection header received
0812 07:00:17.741689 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.741692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.741737 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.741819 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.742461 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.743200 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.744444 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.744649 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.744723 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.746062 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.746066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.746069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.746105 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.747553 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.747960 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.747963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.747964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.748036 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.748307 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.748311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.748312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.748533 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.748660 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.749004 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.749149 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 07:00:17.749763 32245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.741113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.27.110.194:34347)
0812 07:00:17.741418 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:17.741436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:17.741595 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:17.741868 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:17.741874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:17.742148 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:17.742320 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.742331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.743345 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.743353 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.744320 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.744336 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.744774 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.744780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.744909 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.745380 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:17.745398 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:17.745905 (+ 507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:17.747663 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.747669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.747672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.747872 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.748122 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.748124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.748125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.748200 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.748661 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:17.748665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:17.748792 (+ 127us) client_negotiation.cc:769] Sending connection context
0812 07:00:17.749463 (+ 671us) client_negotiation.cc:240] Negotiation successful
0812 07:00:17.749620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:17.765640 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39245
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:17.878830 32250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:17.879055 32250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:17.879197 32250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:17.883157 32250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:17.883208 32250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:17.883277 32250 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 07:00:17.883312 32250 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:17.883390 32250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:17.888470 32250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:39245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:17.889505 32250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:17.890524 32250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:17.900422 32260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.900432 32264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:17.900440 32258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:17.900832 32250 server_base.cc:1034] running on GCE node
I20260812 07:00:17.901196 32250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:17.901839 32250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:17.903048 32250 hybrid_clock.cc:648] HybridClock initialized: now 1786518017903023 us; error 53 us; skew 500 ppm
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:17.905859 32250 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:17.906716 32250 webserver.cc:460] Webserver started at http://127.27.110.195:38839/ using document root <none> and password file <none>
I20260812 07:00:17.907347 32250 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:17.907441 32250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:17.907719 32250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:17.909734 32250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "55fc7aa0cec14a04b4f0a13076881767"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "e68afb2adc2f2dc767235e98275d106b"
server_key_iv: "2adabf2d98d3250af579d5a2361d530e"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.910284 32250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "55fc7aa0cec14a04b4f0a13076881767"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "e68afb2adc2f2dc767235e98275d106b"
server_key_iv: "2adabf2d98d3250af579d5a2361d530e"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.913978 32250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260812 07:00:17.916527 32279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.917868 32250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:17.918023 32250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "55fc7aa0cec14a04b4f0a13076881767"
format_stamp: "Formatted at 2026-08-12 07:00:17 on dist-test-slave-w4v5"
server_key: "e68afb2adc2f2dc767235e98275d106b"
server_key_iv: "2adabf2d98d3250af579d5a2361d530e"
server_key_version: "encryptionkey@0"
I20260812 07:00:17.918151 32250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:17.936164 32250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:17.939621 32250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:17.939821 32250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:17.941318 32250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:17.941370 32250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:17.941434 32250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:17.941468 32250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:17.952425 32250 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:32829
I20260812 07:00:17.952504 32550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:32829 every 8 connection(s)
I20260812 07:00:17.953316 32250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:17.956583 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.942122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:32875 (local address 127.27.110.254:39245)
0812 07:00:17.942256 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.942262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.943216 (+ 954us) server_negotiation.cc:407] Connection header received
0812 07:00:17.944392 (+ 1176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.944399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.944650 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.944759 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.946674 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.947583 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.949447 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.949664 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.949747 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.952880 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.952883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.952887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.952919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.954540 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.955067 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.955070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.955134 (+ 64us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.955235 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.955585 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.955588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.955589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.955767 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.955846 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.956301 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.956457 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 07:00:17.957297 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.942586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.27.110.195:32875)
0812 07:00:17.943031 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:17.943106 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:17.944145 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:17.945126 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:17.945136 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:17.945649 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:17.946415 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.946438 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.947738 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.947743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.949317 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.949328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.949829 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.949833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.950055 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.950756 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:17.950779 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:17.952712 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:17.954669 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.954674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.954684 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.954949 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.955350 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.955353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.955355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.955490 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.955864 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:17.955868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:17.956078 (+ 210us) client_negotiation.cc:769] Sending connection context
0812 07:00:17.956239 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 07:00:17.956470 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":137,"threads_started":1}
I20260812 07:00:17.958532 32551 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39245
I20260812 07:00:17.958706 32551 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:17.959265 32551 heartbeater.cc:511] Master 127.27.110.254:39245 requested a full tablet report, sending...
I20260812 07:00:17.960283 31497 ts_manager.cc:194] Registered new tserver with Master: 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829)
I20260812 07:00:17.960952 31497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:32875
I20260812 07:00:17.962606 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 32250
I20260812 07:00:17.962746 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:17.963063 28091 external_mini_cluster.cc:1428] Setting key cca0d100f60507ed4d0974b20d773a41
I20260812 07:00:17.965211 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:00:17.968549 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.960922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:37799 (local address 127.27.110.254:39245)
0812 07:00:17.961063 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.961068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.961343 (+ 275us) server_negotiation.cc:407] Connection header received
0812 07:00:17.961449 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.961452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.961496 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.961587 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.962046 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.962641 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.963681 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.963913 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.964014 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.965179 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.965182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.965185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.965209 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.966820 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.967292 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.967298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.967303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.967398 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.967689 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.967695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.967700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.967937 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.968044 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.968354 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.968443 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:00:17.968677 32563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.960839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.27.110.195:37799)
0812 07:00:17.961252 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:17.961267 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:17.961362 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:17.961616 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:17.961619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:17.961829 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:17.961935 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.961942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.962803 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.962809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.963558 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:17.963567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.964026 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.964029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:17.964126 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.964620 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:17.964635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:17.965032 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:17.966933 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.966935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.966937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.967198 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.967515 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:17.967517 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:17.967518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:17.967596 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:17.968065 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:17.968071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:17.968179 (+ 108us) client_negotiation.cc:769] Sending connection context
0812 07:00:17.968301 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 07:00:17.968458 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":101,"threads_started":1}
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:00:17 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:17.993659 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:17.985213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.27.110.254:39245)
0812 07:00:17.985382 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:17.985386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:17.985400 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:17.985440 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:17.985442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:17.985481 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:17.985553 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:17.986253 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.986798 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.987679 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:17.987894 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:17.987986 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:17.990505 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:17.990509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:17.990512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:17.990539 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:17.991905 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.992419 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.992422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.992424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.992507 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:17.992786 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:17.992790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:17.992792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:17.992973 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:17.993123 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:17.993361 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 07:00:17.993478 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:00:17.996742 31497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:17.999919 31497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:18.014911 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.009498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48702 (local address 127.27.110.194:40359)
0812 07:00:18.010126 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.010137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.010166 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:00:18.010246 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.010251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.010421 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.010609 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:18.010877 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.011818 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.013290 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.014181 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.014329 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.014466 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.014645 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.014751 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":133,"threads_started":1}
I20260812 07:00:18.014912 32590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.008844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:40359 (local address 127.0.0.1:48702)
0812 07:00:18.009856 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.009868 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.009934 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.010598 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.010604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.010626 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:18.010769 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.010776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.011938 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.011942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.013168 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.013177 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.014311 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.014315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.014429 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.014472 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.014606 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.014737 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":40,"thread_start_us":113,"threads_started":1}
I20260812 07:00:18.015360 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.009077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33345 (local address 127.0.0.1:43876)
0812 07:00:18.010115 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.010130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.010220 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.010775 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.010780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.010796 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:18.010903 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.010908 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.012361 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.012364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.013622 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.013633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.014880 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.014884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.014988 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.015020 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.015139 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.015221 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260812 07:00:18.015864 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.009604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.27.110.193:33345)
0812 07:00:18.009928 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.009935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.010205 (+ 270us) server_negotiation.cc:407] Connection header received
0812 07:00:18.010333 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.010340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.010560 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.010700 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:18.011029 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.012208 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.013753 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.014541 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.014692 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.014813 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.015574 (+ 761us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.015696 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":67,"threads_started":1}
I20260812 07:00:18.018256 32099 tablet_service.cc:1467] Processing CreateTablet for tablet 095d382cf3d44ae68d1baac8acc94c45 (DEFAULT_TABLE table=test-table [id=60dd3ff6546b40c19b186c9f54ddd1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:18.018465 31793 tablet_service.cc:1467] Processing CreateTablet for tablet 095d382cf3d44ae68d1baac8acc94c45 (DEFAULT_TABLE table=test-table [id=60dd3ff6546b40c19b186c9f54ddd1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:18.019490 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.008671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.0.0.1:51522)
0812 07:00:18.009730 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.009783 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.009913 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.011048 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.011051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.011093 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:18.011191 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.011197 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.012540 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.012544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.013769 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.013776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.015078 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.015082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.019153 (+ 4071us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.019188 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.019271 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.019356 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260812 07:00:18.019488 32099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095d382cf3d44ae68d1baac8acc94c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.019490 31793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095d382cf3d44ae68d1baac8acc94c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.020005 32593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.008841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.27.110.195:32829)
0812 07:00:18.010487 (+ 1646us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.010499 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.010519 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:00:18.010590 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.010594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.010766 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.010904 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:18.011323 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.012407 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.013985 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.014733 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.014875 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.015007 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.019743 (+ 4736us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.019857 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":112,"threads_started":1}
I20260812 07:00:18.022709 32392 tablet_service.cc:1467] Processing CreateTablet for tablet 095d382cf3d44ae68d1baac8acc94c45 (DEFAULT_TABLE table=test-table [id=60dd3ff6546b40c19b186c9f54ddd1e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:18.023789 32392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095d382cf3d44ae68d1baac8acc94c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.026306 32598 tablet_bootstrap.cc:492] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Bootstrap starting.
I20260812 07:00:18.027822 32602 tablet_bootstrap.cc:492] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Bootstrap starting.
I20260812 07:00:18.028530 32598 tablet_bootstrap.cc:654] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.029294 32598 log.cc:824] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:18.030000 32602 tablet_bootstrap.cc:654] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.030872 32602 log.cc:824] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:18.031368 32598 tablet_bootstrap.cc:492] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: No bootstrap required, opened a new log
I20260812 07:00:18.031589 32598 ts_tablet_manager.cc:1397] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260812 07:00:18.032783 32609 tablet_bootstrap.cc:492] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Bootstrap starting.
I20260812 07:00:18.033013 32602 tablet_bootstrap.cc:492] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: No bootstrap required, opened a new log
I20260812 07:00:18.033200 32602 ts_tablet_manager.cc:1397] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:00:18.034575 32609 tablet_bootstrap.cc:654] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.035336 32609 log.cc:824] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:18.035843 32598 raft_consensus.cc:348] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.036131 32598 raft_consensus.cc:374] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.036191 32598 raft_consensus.cc:729] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491ae8a46ae46f49202ea9834c31206, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.036648 32598 consensus_queue.cc:260] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.037242 32609 tablet_bootstrap.cc:492] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: No bootstrap required, opened a new log
I20260812 07:00:18.037421 32609 ts_tablet_manager.cc:1397] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 07:00:18.037741 32598 ts_tablet_manager.cc:1428] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:00:18.038122 32232 heartbeater.cc:503] Master 127.27.110.254:39245 was elected leader, sending a full tablet report...
I20260812 07:00:18.037945 32602 raft_consensus.cc:348] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.038203 32602 raft_consensus.cc:374] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.038304 32602 raft_consensus.cc:729] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ba4042f0e47bab53f2b6f366085c5, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.038761 32602 consensus_queue.cc:260] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.039742 32602 ts_tablet_manager.cc:1428] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 07:00:18.040112 31938 heartbeater.cc:503] Master 127.27.110.254:39245 was elected leader, sending a full tablet report...
I20260812 07:00:18.041353 32609 raft_consensus.cc:348] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.041625 32609 raft_consensus.cc:374] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.041688 32609 raft_consensus.cc:729] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.042085 32609 consensus_queue.cc:260] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.042822 32551 heartbeater.cc:503] Master 127.27.110.254:39245 was elected leader, sending a full tablet report...
I20260812 07:00:18.043143 32609 ts_tablet_manager.cc:1428] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260812 07:00:18.204317 32554 tablet.cc:2369] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:00:18.234995 32235 tablet.cc:2369] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:00:18.242626 31939 tablet.cc:2369] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:18.322968 32613 raft_consensus.cc:482] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:18.323235 32613 raft_consensus.cc:504] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.324443 32613 leader_election.cc:288] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359), 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345)
I20260812 07:00:18.328984 32291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.324726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33345 (local address 127.27.110.195:41987)
0812 07:00:18.324852 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.324869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.324971 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.325255 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.325257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.325277 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:18.325427 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.325436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.326523 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.326527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.327581 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.327590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.328484 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.328488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.328655 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.328682 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.328762 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.328844 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 07:00:18.328953 32597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.324846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:41987 (local address 127.27.110.193:33345)
0812 07:00:18.324974 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.324979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.325005 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:00:18.325074 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.325078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.325136 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.325225 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:18.325572 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.326415 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.327720 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.328368 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.328458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.328504 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.328786 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.328850 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 07:00:18.329515 32595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.324849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:39361 (local address 127.27.110.194:40359)
0812 07:00:18.325014 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.325019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.325390 (+ 371us) server_negotiation.cc:407] Connection header received
0812 07:00:18.325476 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.325482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.325542 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.325612 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:18.326342 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.327053 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.328334 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.328915 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.329036 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.329087 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.329314 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.329388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 07:00:18.329550 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.324737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:40359 (local address 127.27.110.195:39361)
0812 07:00:18.325277 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.325292 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.325395 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.325651 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.325655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.325673 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:18.326223 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.326232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.327183 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.327186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.328204 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.328215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.329076 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.329080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.329188 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.329217 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.329291 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.329352 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":168,"thread_start_us":120,"threads_started":1}
I20260812 07:00:18.329775 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d382cf3d44ae68d1baac8acc94c45" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5" is_pre_election: true
I20260812 07:00:18.330114 31834 raft_consensus.cc:2454] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 0.
I20260812 07:00:18.330271 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d382cf3d44ae68d1baac8acc94c45" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206" is_pre_election: true
I20260812 07:00:18.330602 32140 raft_consensus.cc:2454] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 0.
I20260812 07:00:18.330673 32285 leader_election.cc:302] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 55fc7aa0cec14a04b4f0a13076881767; no voters:
I20260812 07:00:18.330972 32613 raft_consensus.cc:2781] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:18.331099 32613 raft_consensus.cc:482] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:18.331166 32613 raft_consensus.cc:3037] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.332288 32613 raft_consensus.cc:504] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.332644 32613 leader_election.cc:288] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 election: Requested vote from peers 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359), 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345)
I20260812 07:00:18.333034 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d382cf3d44ae68d1baac8acc94c45" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206"
I20260812 07:00:18.333086 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095d382cf3d44ae68d1baac8acc94c45" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5"
I20260812 07:00:18.333168 32140 raft_consensus.cc:3037] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.333194 31834 raft_consensus.cc:3037] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.334457 31834 raft_consensus.cc:2454] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 1.
I20260812 07:00:18.334457 32140 raft_consensus.cc:2454] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 1.
I20260812 07:00:18.334761 32285 leader_election.cc:302] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 55fc7aa0cec14a04b4f0a13076881767, 8491ae8a46ae46f49202ea9834c31206; no voters:
I20260812 07:00:18.334904 32613 raft_consensus.cc:2781] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:18.335175 32613 raft_consensus.cc:686] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Running, Role: LEADER
I20260812 07:00:18.335506 32613 consensus_queue.cc:237] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } }
I20260812 07:00:18.338748 31497 catalog_manager.cc:5495] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195). New cstate: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:00:18 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518017, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:18.405865 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.397082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40424 (local address 127.27.110.254:39245)
0812 07:00:18.397257 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.397263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.397457 (+ 194us) server_negotiation.cc:407] Connection header received
0812 07:00:18.397627 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.397630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.397676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.397764 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:18.398495 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.399023 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.399917 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.400136 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.400251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.402622 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:18.402625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:18.402628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:18.402658 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:18.404094 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:18.404575 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:18.404580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:18.404582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:18.404662 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:18.404941 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:18.404946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:18.404948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:18.405119 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:18.405254 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.405545 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.405706 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":61}
I20260812 07:00:18.405865 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.397038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39245 (local address 127.0.0.1:40424)
0812 07:00:18.397367 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.397380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.397518 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.397788 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.397791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.398270 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:18.398370 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.398377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.399181 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.399186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.399793 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.399802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.400257 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.400261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.400393 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.401012 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:18.401025 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:18.402485 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:18.404215 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:18.404225 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:18.404234 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:18.404471 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:18.404764 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:18.404767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:18.404769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:18.404851 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:18.405254 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:18.405263 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:18.405376 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.405497 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.405653 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":4,"thread_start_us":133,"threads_started":1}
I20260812 07:00:18.409881 31497 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:40424:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 07:00:18.410609 31497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:18.415503 31793 tablet_service.cc:1467] Processing CreateTablet for tablet b89257aa7bcd4b4f852254e99c990d2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ab87c688e25466481549a7e1a5ef147]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:18.415580 32099 tablet_service.cc:1467] Processing CreateTablet for tablet b89257aa7bcd4b4f852254e99c990d2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ab87c688e25466481549a7e1a5ef147]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:18.415746 32392 tablet_service.cc:1467] Processing CreateTablet for tablet b89257aa7bcd4b4f852254e99c990d2a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9ab87c688e25466481549a7e1a5ef147]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:18.416297 31793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89257aa7bcd4b4f852254e99c990d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.416297 32099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89257aa7bcd4b4f852254e99c990d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.416512 32392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89257aa7bcd4b4f852254e99c990d2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.419409 32602 tablet_bootstrap.cc:492] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: Bootstrap starting.
I20260812 07:00:18.419414 32598 tablet_bootstrap.cc:492] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: Bootstrap starting.
I20260812 07:00:18.419414 32609 tablet_bootstrap.cc:492] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: Bootstrap starting.
I20260812 07:00:18.420288 32602 tablet_bootstrap.cc:654] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.420389 32609 tablet_bootstrap.cc:654] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.420389 32598 tablet_bootstrap.cc:654] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.421831 32602 tablet_bootstrap.cc:492] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: No bootstrap required, opened a new log
I20260812 07:00:18.421890 32598 tablet_bootstrap.cc:492] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: No bootstrap required, opened a new log
I20260812 07:00:18.421931 32602 ts_tablet_manager.cc:1397] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:18.421958 32598 ts_tablet_manager.cc:1397] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 07:00:18.422211 32609 tablet_bootstrap.cc:492] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: No bootstrap required, opened a new log
I20260812 07:00:18.422314 32609 ts_tablet_manager.cc:1397] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:18.422420 32598 raft_consensus.cc:348] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.422420 32602 raft_consensus.cc:348] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.422546 32598 raft_consensus.cc:374] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.422546 32602 raft_consensus.cc:374] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.422591 32602 raft_consensus.cc:729] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ba4042f0e47bab53f2b6f366085c5, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.422591 32598 raft_consensus.cc:729] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491ae8a46ae46f49202ea9834c31206, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.422721 32602 consensus_queue.cc:260] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.422775 32609 raft_consensus.cc:348] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.422878 32609 raft_consensus.cc:374] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.422905 32609 raft_consensus.cc:729] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.423007 32602 ts_tablet_manager.cc:1428] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:18.422974 32612 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.423035 32609 consensus_queue.cc:260] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.423141 32612 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.423372 32609 ts_tablet_manager.cc:1428] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:18.423333 32613 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.423451 32613 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.422789 32598 consensus_queue.cc:260] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.423892 32598 ts_tablet_manager.cc:1428] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:18.424220 32611 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.424432 32611 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.619179 32613 raft_consensus.cc:482] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:18.619318 32613 raft_consensus.cc:504] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.619819 32613 leader_election.cc:288] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345), 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359)
I20260812 07:00:18.620306 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89257aa7bcd4b4f852254e99c990d2a" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206" is_pre_election: true
I20260812 07:00:18.620319 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89257aa7bcd4b4f852254e99c990d2a" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5" is_pre_election: true
I20260812 07:00:18.620481 31834 raft_consensus.cc:2454] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 0.
I20260812 07:00:18.620483 32140 raft_consensus.cc:2454] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 0.
I20260812 07:00:18.620834 32281 leader_election.cc:302] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fc7aa0cec14a04b4f0a13076881767, 8491ae8a46ae46f49202ea9834c31206; no voters:
I20260812 07:00:18.621028 32613 raft_consensus.cc:2781] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:18.621098 32613 raft_consensus.cc:482] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:18.621133 32613 raft_consensus.cc:3037] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.622236 32613 raft_consensus.cc:504] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.622536 32613 leader_election.cc:288] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 election: Requested vote from peers 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345), 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359)
I20260812 07:00:18.622906 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89257aa7bcd4b4f852254e99c990d2a" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5"
I20260812 07:00:18.622934 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89257aa7bcd4b4f852254e99c990d2a" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206"
I20260812 07:00:18.623018 31834 raft_consensus.cc:3037] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.623054 32140 raft_consensus.cc:3037] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.623981 31834 raft_consensus.cc:2454] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 1.
I20260812 07:00:18.624125 32140 raft_consensus.cc:2454] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 1.
I20260812 07:00:18.624302 32285 leader_election.cc:302] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 55fc7aa0cec14a04b4f0a13076881767; no voters:
I20260812 07:00:18.624554 32613 raft_consensus.cc:2781] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:18.624655 32613 raft_consensus.cc:686] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Running, Role: LEADER
I20260812 07:00:18.624800 32613 consensus_queue.cc:237] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.625437 32613 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: TxnStatusTablet state changed. Reason: New leader 55fc7aa0cec14a04b4f0a13076881767. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.625540 32613 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:18.625811 32634 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 07:00:18.626061 32634 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 07:00:18.626585 31497 catalog_manager.cc:5495] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195). New cstate: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } health_report { overall_health: HEALTHY } } }
I20260812 07:00:18.657378 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.653375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.0.0.1:51524)
0812 07:00:18.653644 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.653662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.653771 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.654143 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.654146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.654154 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:18.654282 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.654287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.655093 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.655098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.655720 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.655728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.656219 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.656222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.656315 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.656359 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:18.657059 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:18.657076 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.657146 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.657237 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260812 07:00:18.657401 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.653497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.27.110.195:32829)
0812 07:00:18.653751 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.653756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.653770 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:18.653952 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.653955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.654009 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.654145 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:18.654394 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.654951 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.655843 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.656086 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.656205 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.656537 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:18.656956 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:18.657089 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.657157 (+ 68us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.657213 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260812 07:00:18.663194 31834 raft_consensus.cc:1261] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Refusing update from remote peer 55fc7aa0cec14a04b4f0a13076881767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:18.663192 32140 raft_consensus.cc:1261] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Refusing update from remote peer 55fc7aa0cec14a04b4f0a13076881767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:18.663931 32615 consensus_queue.cc:1035] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:18.664139 32613 consensus_queue.cc:1035] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:18.667801 32612 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: TxnStatusTablet state changed. Reason: New leader 55fc7aa0cec14a04b4f0a13076881767. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.667927 32612 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.668390 32611 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: TxnStatusTablet state changed. Reason: New leader 55fc7aa0cec14a04b4f0a13076881767. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.668515 32611 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.672551 32613 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.672686 32613 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:18.672945 32612 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.673055 32612 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 083ba4042f0e47bab53f2b6f366085c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.673597 32639 mvcc.cc:205] Tried to move back new op lower bound from 7317577804436951040 to 7317577804288593920. Current Snapshot: MvccSnapshot[applied={T|T < 7317577804436951040}]
I20260812 07:00:18.673751 32641 mvcc.cc:205] Tried to move back new op lower bound from 7317577804436951040 to 7317577804288593920. Current Snapshot: MvccSnapshot[applied={T|T < 7317577804436951040}]
I20260812 07:00:18.673875 32615 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.673979 32615 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 55fc7aa0cec14a04b4f0a13076881767: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:18.675762 32611 tablet_replica.cc:439] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } } }
I20260812 07:00:18.675871 32611 tablet_replica.cc:442] T b89257aa7bcd4b4f852254e99c990d2a P 8491ae8a46ae46f49202ea9834c31206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:18.676671 32640 mvcc.cc:205] Tried to move back new op lower bound from 7317577804436951040 to 7317577804288593920. Current Snapshot: MvccSnapshot[applied={T|T < 7317577804436951040}]
I20260812 07:00:18.685915 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.681774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.27.110.195:32829)
0812 07:00:18.681956 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.681960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.682199 (+ 239us) server_negotiation.cc:407] Connection header received
0812 07:00:18.682315 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.682318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.682362 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.682466 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:18.682771 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.683644 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.684904 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.685121 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.685207 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.685497 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:18.685564 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:18.685625 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.685726 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.685771 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:00:18.698570 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.694030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:58355 (local address 127.27.110.195:32829)
0812 07:00:18.694166 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:18.694170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:18.694382 (+ 212us) server_negotiation.cc:407] Connection header received
0812 07:00:18.694514 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:18.694516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:18.694557 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:18.694622 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:18.695453 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.695951 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.697348 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.697650 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.697744 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.698113 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:18.698186 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:18.698244 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:18.698364 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 07:00:18.698425 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":56}
I20260812 07:00:18.698711 32677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:18.693965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.27.110.195:58355)
0812 07:00:18.694285 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:18.694300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:18.694423 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:18.694658 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:18.694661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:18.694673 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:18.694798 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.694804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.696117 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.696121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.697172 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:18.697185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:18.697774 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:18.697778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:18.697882 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:18.697941 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:18.698274 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:18.698281 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:00:18.698356 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 07:00:18.698471 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":18,"thread_start_us":130,"threads_started":1}
I20260812 07:00:18.706749 31834 raft_consensus.cc:1261] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Refusing update from remote peer 55fc7aa0cec14a04b4f0a13076881767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:18.706861 32140 raft_consensus.cc:1261] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Refusing update from remote peer 55fc7aa0cec14a04b4f0a13076881767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:18.707250 32613 consensus_queue.cc:1035] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:18.707443 32653 consensus_queue.cc:1035] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:18.765272 31494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:18.765519 31494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:18.769250 31494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Sending DeleteTablet for 3 replicas of tablet 095d382cf3d44ae68d1baac8acc94c45
I20260812 07:00:18.770227 32099 tablet_service.cc:1514] Processing DeleteTablet for tablet 095d382cf3d44ae68d1baac8acc94c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:18 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48702
I20260812 07:00:18.770293 31793 tablet_service.cc:1514] Processing DeleteTablet for tablet 095d382cf3d44ae68d1baac8acc94c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:18 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43876
I20260812 07:00:18.770414 32391 tablet_service.cc:1514] Processing DeleteTablet for tablet 095d382cf3d44ae68d1baac8acc94c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:18 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:51522
I20260812 07:00:18.770931 32688 tablet_replica.cc:330] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: stopping tablet replica
I20260812 07:00:18.771188 32684 tablet_replica.cc:330] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: stopping tablet replica
I20260812 07:00:18.771332 32688 raft_consensus.cc:2229] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:18.771634 32684 raft_consensus.cc:2229] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:18.771772 32688 raft_consensus.cc:2258] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:18.771993 32684 raft_consensus.cc:2258] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:18.771993 31494 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:18.772467 31494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:18.773335 32688 ts_tablet_manager.cc:1905] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:18.773736 32684 ts_tablet_manager.cc:1905] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:18.776906 32688 ts_tablet_manager.cc:1918] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:18.777036 32688 log.cc:1196] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/095d382cf3d44ae68d1baac8acc94c45
I20260812 07:00:18.777060 32684 ts_tablet_manager.cc:1918] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:18.777294 32684 log.cc:1196] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/095d382cf3d44ae68d1baac8acc94c45
I20260812 07:00:18.777489 32688 ts_tablet_manager.cc:1939] T 095d382cf3d44ae68d1baac8acc94c45 P 55fc7aa0cec14a04b4f0a13076881767: Deleting consensus metadata
I20260812 07:00:18.777808 32684 ts_tablet_manager.cc:1939] T 095d382cf3d44ae68d1baac8acc94c45 P 083ba4042f0e47bab53f2b6f366085c5: Deleting consensus metadata
I20260812 07:00:18.778373 31793 tablet_service.cc:1467] Processing CreateTablet for tablet 4e673a1932b2450e9bba31287c087a25 (DEFAULT_TABLE table=test-table [id=3ed6ffcc4eca4f00ba02a6ae54b305a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:18.778688 31793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e673a1932b2450e9bba31287c087a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.779328 31469 catalog_manager.cc:4841] TS 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829): tablet 095d382cf3d44ae68d1baac8acc94c45 (table test-table [id=60dd3ff6546b40c19b186c9f54ddd1e7]) successfully deleted
I20260812 07:00:18.780124 32685 tablet_replica.cc:330] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: stopping tablet replica
I20260812 07:00:18.780500 32685 raft_consensus.cc:2229] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:18.780747 32685 raft_consensus.cc:2258] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:18.781394 31474 catalog_manager.cc:4841] TS 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345): tablet 095d382cf3d44ae68d1baac8acc94c45 (table test-table [id=60dd3ff6546b40c19b186c9f54ddd1e7]) successfully deleted
I20260812 07:00:18.781947 32099 tablet_service.cc:1467] Processing CreateTablet for tablet 4e673a1932b2450e9bba31287c087a25 (DEFAULT_TABLE table=test-table [id=3ed6ffcc4eca4f00ba02a6ae54b305a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:18.782199 32685 ts_tablet_manager.cc:1905] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:18.782240 32099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e673a1932b2450e9bba31287c087a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.783389 32391 tablet_service.cc:1467] Processing CreateTablet for tablet 4e673a1932b2450e9bba31287c087a25 (DEFAULT_TABLE table=test-table [id=3ed6ffcc4eca4f00ba02a6ae54b305a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:18.783707 32391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e673a1932b2450e9bba31287c087a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:18.785526 32598 tablet_bootstrap.cc:492] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Bootstrap starting.
I20260812 07:00:18.785676 32685 ts_tablet_manager.cc:1918] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:18.785761 32685 log.cc:1196] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/095d382cf3d44ae68d1baac8acc94c45
I20260812 07:00:18.786229 32609 tablet_bootstrap.cc:492] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Bootstrap starting.
I20260812 07:00:18.787093 32602 tablet_bootstrap.cc:492] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Bootstrap starting.
I20260812 07:00:18.787300 32609 tablet_bootstrap.cc:654] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.787297 32685 ts_tablet_manager.cc:1939] T 095d382cf3d44ae68d1baac8acc94c45 P 8491ae8a46ae46f49202ea9834c31206: Deleting consensus metadata
I20260812 07:00:18.788213 32602 tablet_bootstrap.cc:654] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.788928 31468 catalog_manager.cc:4841] TS 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359): tablet 095d382cf3d44ae68d1baac8acc94c45 (table test-table [id=60dd3ff6546b40c19b186c9f54ddd1e7]) successfully deleted
I20260812 07:00:18.789353 32598 tablet_bootstrap.cc:654] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:18.790345 32602 tablet_bootstrap.cc:492] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: No bootstrap required, opened a new log
I20260812 07:00:18.790424 32602 ts_tablet_manager.cc:1397] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:18.790593 32598 tablet_bootstrap.cc:492] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: No bootstrap required, opened a new log
I20260812 07:00:18.790666 32598 ts_tablet_manager.cc:1397] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260812 07:00:18.790992 32602 raft_consensus.cc:348] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.791159 32602 raft_consensus.cc:374] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.791208 32602 raft_consensus.cc:729] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ba4042f0e47bab53f2b6f366085c5, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.791180 32598 raft_consensus.cc:348] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.791288 32598 raft_consensus.cc:374] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.791317 32598 raft_consensus.cc:729] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491ae8a46ae46f49202ea9834c31206, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.791440 32598 consensus_queue.cc:260] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.791694 32598 ts_tablet_manager.cc:1428] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:18.791365 32602 consensus_queue.cc:260] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.792994 32602 ts_tablet_manager.cc:1428] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:18.794646 32609 tablet_bootstrap.cc:492] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: No bootstrap required, opened a new log
I20260812 07:00:18.794775 32609 ts_tablet_manager.cc:1397] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260812 07:00:18.795457 32609 raft_consensus.cc:348] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.795655 32609 raft_consensus.cc:374] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:18.795720 32609 raft_consensus.cc:729] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Initialized, Role: FOLLOWER
I20260812 07:00:18.795878 32609 consensus_queue.cc:260] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.796197 32609 ts_tablet_manager.cc:1428] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 07:00:18.884617 32653 raft_consensus.cc:482] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:18.884768 32653 raft_consensus.cc:504] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.885258 32653 leader_election.cc:288] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345), 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359)
I20260812 07:00:18.885739 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e673a1932b2450e9bba31287c087a25" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5" is_pre_election: true
I20260812 07:00:18.885757 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e673a1932b2450e9bba31287c087a25" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206" is_pre_election: true
I20260812 07:00:18.885917 31834 raft_consensus.cc:2454] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 0.
I20260812 07:00:18.885928 32140 raft_consensus.cc:2454] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 0.
I20260812 07:00:18.886272 32285 leader_election.cc:302] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 55fc7aa0cec14a04b4f0a13076881767; no voters:
I20260812 07:00:18.886471 32653 raft_consensus.cc:2781] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:18.886559 32653 raft_consensus.cc:482] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:18.886612 32653 raft_consensus.cc:3037] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.887784 32653 raft_consensus.cc:504] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.888162 32653 leader_election.cc:288] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 election: Requested vote from peers 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345), 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359)
I20260812 07:00:18.888516 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e673a1932b2450e9bba31287c087a25" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5"
I20260812 07:00:18.888648 31834 raft_consensus.cc:3037] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.889770 31834 raft_consensus.cc:2454] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 1.
I20260812 07:00:18.890026 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e673a1932b2450e9bba31287c087a25" candidate_uuid: "55fc7aa0cec14a04b4f0a13076881767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206"
I20260812 07:00:18.890085 32285 leader_election.cc:302] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 55fc7aa0cec14a04b4f0a13076881767; no voters:
I20260812 07:00:18.890163 32140 raft_consensus.cc:3037] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:18.890287 32653 raft_consensus.cc:2781] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:18.890388 32653 raft_consensus.cc:686] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 LEADER]: Becoming Leader. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Running, Role: LEADER
I20260812 07:00:18.890558 32653 consensus_queue.cc:237] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:18.891383 32140 raft_consensus.cc:2454] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55fc7aa0cec14a04b4f0a13076881767 in term 1.
I20260812 07:00:18.892558 31497 catalog_manager.cc:5495] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195). New cstate: current_term: 1 leader_uuid: "55fc7aa0cec14a04b4f0a13076881767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } health_report { overall_health: HEALTHY } } }
I20260812 07:00:18.912400 31834 raft_consensus.cc:1261] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Refusing update from remote peer 55fc7aa0cec14a04b4f0a13076881767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:18.912400 32140 raft_consensus.cc:1261] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Refusing update from remote peer 55fc7aa0cec14a04b4f0a13076881767: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:18.912833 32615 consensus_queue.cc:1035] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:18.913098 32615 consensus_queue.cc:1035] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:00:18.928815 32639 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 07:00:20.494838 31497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:20.495012 31497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:20.498050 31497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Sending DeleteTablet for 3 replicas of tablet 4e673a1932b2450e9bba31287c087a25
I20260812 07:00:20.498639 31793 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e673a1932b2450e9bba31287c087a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:20 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43876
I20260812 07:00:20.498678 32099 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e673a1932b2450e9bba31287c087a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:20 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48702
I20260812 07:00:20.498771 32392 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e673a1932b2450e9bba31287c087a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:20 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:51522
I20260812 07:00:20.499043 32742 tablet_replica.cc:330] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: stopping tablet replica
I20260812 07:00:20.499130 32743 tablet_replica.cc:330] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: stopping tablet replica
I20260812 07:00:20.499270 32742 raft_consensus.cc:2229] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:20.499285 32743 raft_consensus.cc:2229] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:20.499435 32742 raft_consensus.cc:2258] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:20.499701 32743 raft_consensus.cc:2258] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:20.500206 32742 ts_tablet_manager.cc:1905] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:20.500438 32743 ts_tablet_manager.cc:1905] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:20.503273 32743 ts_tablet_manager.cc:1918] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:00:20.503273 32742 ts_tablet_manager.cc:1918] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:00:20.503360 32742 log.cc:1196] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/4e673a1932b2450e9bba31287c087a25
I20260812 07:00:20.503360 32743 log.cc:1196] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/4e673a1932b2450e9bba31287c087a25
I20260812 07:00:20.503683 32742 ts_tablet_manager.cc:1939] T 4e673a1932b2450e9bba31287c087a25 P 8491ae8a46ae46f49202ea9834c31206: Deleting consensus metadata
I20260812 07:00:20.503679 32743 ts_tablet_manager.cc:1939] T 4e673a1932b2450e9bba31287c087a25 P 55fc7aa0cec14a04b4f0a13076881767: Deleting consensus metadata
I20260812 07:00:20.504385 31468 catalog_manager.cc:4841] TS 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359): tablet 4e673a1932b2450e9bba31287c087a25 (table test-table [id=3ed6ffcc4eca4f00ba02a6ae54b305a2]) successfully deleted
I20260812 07:00:20.504505 31469 catalog_manager.cc:4841] TS 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829): tablet 4e673a1932b2450e9bba31287c087a25 (table test-table [id=3ed6ffcc4eca4f00ba02a6ae54b305a2]) successfully deleted
I20260812 07:00:20.507521 32741 tablet_replica.cc:330] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: stopping tablet replica
I20260812 07:00:20.507682 32741 raft_consensus.cc:2229] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:20.507825 32741 raft_consensus.cc:2258] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:20.508404 32741 ts_tablet_manager.cc:1905] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:20.510519 32741 ts_tablet_manager.cc:1918] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:00:20.510615 32741 log.cc:1196] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/4e673a1932b2450e9bba31287c087a25
I20260812 07:00:20.510878 32741 ts_tablet_manager.cc:1939] T 4e673a1932b2450e9bba31287c087a25 P 083ba4042f0e47bab53f2b6f366085c5: Deleting consensus metadata
I20260812 07:00:20.511617 31474 catalog_manager.cc:4841] TS 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345): tablet 4e673a1932b2450e9bba31287c087a25 (table test-table [id=3ed6ffcc4eca4f00ba02a6ae54b305a2]) successfully deleted
Aug 12 07:00:20 dist-test-slave-w4v5 krb5kdc[31395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:00:20 dist-test-slave-w4v5 krb5kdc[31395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518020, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:20.529722 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.521224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.27.110.254:39245)
0812 07:00:20.521587 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.521594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.521610 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:20.521713 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.521715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.521759 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.521838 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:20.522576 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.523225 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.524053 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.524278 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.524360 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.526674 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:20.526677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:20.526679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:20.526702 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:20.528329 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:20.528694 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:20.528697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:20.528698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:20.528775 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:20.529010 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:20.529012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:20.529013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:20.529190 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:20.529292 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.529490 (+ 198us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.529596 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260812 07:00:20.532269 31497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:20.532721 31497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:20.537439 32099 tablet_service.cc:1467] Processing CreateTablet for tablet 725eb05fa29c4803ab35f05289d93073 (DEFAULT_TABLE table=test-table [id=f626f5aadfc14b098adcb5e602c8c9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:20.537431 31793 tablet_service.cc:1467] Processing CreateTablet for tablet 725eb05fa29c4803ab35f05289d93073 (DEFAULT_TABLE table=test-table [id=f626f5aadfc14b098adcb5e602c8c9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:20.537531 32392 tablet_service.cc:1467] Processing CreateTablet for tablet 725eb05fa29c4803ab35f05289d93073 (DEFAULT_TABLE table=test-table [id=f626f5aadfc14b098adcb5e602c8c9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:20.537760 31793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725eb05fa29c4803ab35f05289d93073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:20.537762 32099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725eb05fa29c4803ab35f05289d93073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:20.537806 32392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725eb05fa29c4803ab35f05289d93073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:20.540100 32760 tablet_bootstrap.cc:492] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Bootstrap starting.
I20260812 07:00:20.540967 32763 tablet_bootstrap.cc:492] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Bootstrap starting.
I20260812 07:00:20.541172 32760 tablet_bootstrap.cc:654] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:20.541793 32763 tablet_bootstrap.cc:654] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:20.542457 32761 tablet_bootstrap.cc:492] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Bootstrap starting.
I20260812 07:00:20.543004 32763 tablet_bootstrap.cc:492] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: No bootstrap required, opened a new log
I20260812 07:00:20.543069 32760 tablet_bootstrap.cc:492] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: No bootstrap required, opened a new log
I20260812 07:00:20.543121 32763 ts_tablet_manager.cc:1397] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:20.543179 32760 ts_tablet_manager.cc:1397] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20260812 07:00:20.543589 32763 raft_consensus.cc:348] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.543633 32760 raft_consensus.cc:348] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.543722 32760 raft_consensus.cc:374] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:20.543718 32763 raft_consensus.cc:374] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:20.543777 32763 raft_consensus.cc:729] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Initialized, Role: FOLLOWER
I20260812 07:00:20.543779 32760 raft_consensus.cc:729] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491ae8a46ae46f49202ea9834c31206, State: Initialized, Role: FOLLOWER
I20260812 07:00:20.543919 32763 consensus_queue.cc:260] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.543926 32760 consensus_queue.cc:260] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.544193 32763 ts_tablet_manager.cc:1428] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 07:00:20.544344 32760 ts_tablet_manager.cc:1428] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:20.545377 32761 tablet_bootstrap.cc:654] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:20.547667 32761 tablet_bootstrap.cc:492] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: No bootstrap required, opened a new log
I20260812 07:00:20.547775 32761 ts_tablet_manager.cc:1397] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 07:00:20.548185 32761 raft_consensus.cc:348] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.548286 32761 raft_consensus.cc:374] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:20.548309 32761 raft_consensus.cc:729] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ba4042f0e47bab53f2b6f366085c5, State: Initialized, Role: FOLLOWER
I20260812 07:00:20.548445 32761 consensus_queue.cc:260] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.548820 32761 ts_tablet_manager.cc:1428] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:20.590945 301 raft_consensus.cc:482] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:20.591113 301 raft_consensus.cc:504] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.592260 301 leader_election.cc:288] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359), 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829)
I20260812 07:00:20.596751 302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.592498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:40359 (local address 127.27.110.193:38039)
0812 07:00:20.592927 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.592943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.593093 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.593407 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.593410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.593427 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:20.593584 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.593591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.594395 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.594400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.595512 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.595523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.596287 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.596294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.596393 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.596415 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.596492 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.596561 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":4,"spinlock_wait_cycles":31104,"thread_start_us":120,"threads_started":1}
I20260812 07:00:20.596781 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.592661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:38039 (local address 127.27.110.194:40359)
0812 07:00:20.592993 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.593005 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.593027 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:00:20.593237 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.593243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.593297 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.593388 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:20.593715 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.594274 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.595669 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.596131 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.596249 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.596292 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.596524 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.596622 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20260812 07:00:20.596863 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.592643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:48597 (local address 127.27.110.195:32829)
0812 07:00:20.592999 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.593007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.593100 (+ 93us) server_negotiation.cc:407] Connection header received
0812 07:00:20.593260 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.593263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.593304 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.593401 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:20.593757 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.594278 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.595633 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.596259 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.596385 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.596437 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.596636 (+ 199us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.596714 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":160,"threads_started":1}
I20260812 07:00:20.596997 303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.592498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.27.110.193:48597)
0812 07:00:20.592990 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.593006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.593126 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.593432 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.593435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.593481 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:20.593609 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.593614 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.594418 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.594426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.595504 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.595515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.596389 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.596392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.596495 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.596520 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.596597 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.596661 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":90,"spinlock_wait_cycles":1792,"thread_start_us":109,"threads_started":1}
I20260812 07:00:20.597506 32435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725eb05fa29c4803ab35f05289d93073" candidate_uuid: "083ba4042f0e47bab53f2b6f366085c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc7aa0cec14a04b4f0a13076881767" is_pre_election: true
I20260812 07:00:20.597607 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725eb05fa29c4803ab35f05289d93073" candidate_uuid: "083ba4042f0e47bab53f2b6f366085c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206" is_pre_election: true
I20260812 07:00:20.597775 32435 raft_consensus.cc:2454] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083ba4042f0e47bab53f2b6f366085c5 in term 0.
I20260812 07:00:20.597800 32140 raft_consensus.cc:2454] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 083ba4042f0e47bab53f2b6f366085c5 in term 0.
I20260812 07:00:20.598214 31697 leader_election.cc:302] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 8491ae8a46ae46f49202ea9834c31206; no voters:
I20260812 07:00:20.598518 301 raft_consensus.cc:2781] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:20.598587 301 raft_consensus.cc:482] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:20.598616 301 raft_consensus.cc:3037] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:20.599701 301 raft_consensus.cc:504] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.600093 301 leader_election.cc:288] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [CANDIDATE]: Term 1 election: Requested vote from peers 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359), 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829)
I20260812 07:00:20.600423 32140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725eb05fa29c4803ab35f05289d93073" candidate_uuid: "083ba4042f0e47bab53f2b6f366085c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491ae8a46ae46f49202ea9834c31206"
I20260812 07:00:20.600481 32435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725eb05fa29c4803ab35f05289d93073" candidate_uuid: "083ba4042f0e47bab53f2b6f366085c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc7aa0cec14a04b4f0a13076881767"
I20260812 07:00:20.600554 32140 raft_consensus.cc:3037] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:20.600616 32435 raft_consensus.cc:3037] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:20.601573 32140 raft_consensus.cc:2454] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083ba4042f0e47bab53f2b6f366085c5 in term 1.
I20260812 07:00:20.601794 32435 raft_consensus.cc:2454] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 083ba4042f0e47bab53f2b6f366085c5 in term 1.
I20260812 07:00:20.601861 31697 leader_election.cc:302] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 8491ae8a46ae46f49202ea9834c31206; no voters:
I20260812 07:00:20.602041 301 raft_consensus.cc:2781] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:20.602276 301 raft_consensus.cc:686] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 LEADER]: Becoming Leader. State: Replica: 083ba4042f0e47bab53f2b6f366085c5, State: Running, Role: LEADER
I20260812 07:00:20.602540 301 consensus_queue.cc:237] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.605049 31494 catalog_manager.cc:5495] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193). New cstate: current_term: 1 leader_uuid: "083ba4042f0e47bab53f2b6f366085c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:20.631958 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.628291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.27.110.193:33345)
0812 07:00:20.628576 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.628584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.628596 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:00:20.628669 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.628671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.628722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.628824 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:20.629053 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.629596 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.630488 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.630735 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.630811 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.631128 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.631596 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.631688 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.631785 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.631831 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":147,"threads_started":1}
I20260812 07:00:20.641835 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.638735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:39343 (local address 127.27.110.195:32829)
0812 07:00:20.638860 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.638864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.638993 (+ 129us) server_negotiation.cc:407] Connection header received
0812 07:00:20.639121 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.639125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.639178 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.639243 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:20.639546 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.640066 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.640892 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.641116 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.641198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.641452 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.641518 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.641579 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.641683 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.641729 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:00:20.641863 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.638619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.27.110.193:39343)
0812 07:00:20.638917 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.638929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.639014 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.639272 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.639274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.639280 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:20.639434 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.639443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.640163 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.640165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.640774 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.640782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.641219 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.641221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.641289 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.641329 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.641599 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.641604 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.641673 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.641736 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260812 07:00:20.647658 32140 raft_consensus.cc:1261] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Refusing update from remote peer 083ba4042f0e47bab53f2b6f366085c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:20.647799 32435 raft_consensus.cc:1261] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Refusing update from remote peer 083ba4042f0e47bab53f2b6f366085c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:20.648188 301 consensus_queue.cc:1035] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:20.648368 309 consensus_queue.cc:1035] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:20.665802 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.662211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:57225 (local address 127.27.110.193:33345)
0812 07:00:20.662335 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.662339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.662521 (+ 182us) server_negotiation.cc:407] Connection header received
0812 07:00:20.662632 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.662636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.662717 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.662808 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:20.663062 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.663730 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.664569 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.664801 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.664931 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.665260 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.665354 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.665438 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.665588 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.665647 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:00:20.665817 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.662155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33345 (local address 127.27.110.195:57225)
0812 07:00:20.662428 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.662444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.662546 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.662811 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.662813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.662818 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:20.662909 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.662913 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.663835 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.663838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.664454 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.664462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.664945 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.664948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.665045 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.665092 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.665475 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.665482 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.665556 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.665651 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260812 07:00:20.688110 31497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:20.688293 31497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:20.690409 31497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Sending DeleteTablet for 3 replicas of tablet 725eb05fa29c4803ab35f05289d93073
I20260812 07:00:20.691021 32099 tablet_service.cc:1514] Processing DeleteTablet for tablet 725eb05fa29c4803ab35f05289d93073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:20 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48702
I20260812 07:00:20.691045 31793 tablet_service.cc:1514] Processing DeleteTablet for tablet 725eb05fa29c4803ab35f05289d93073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:20 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43876
I20260812 07:00:20.691251 32742 tablet_replica.cc:330] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: stopping tablet replica
I20260812 07:00:20.691273 32741 tablet_replica.cc:330] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: stopping tablet replica
I20260812 07:00:20.691376 32742 raft_consensus.cc:2229] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:20.691396 32741 raft_consensus.cc:2229] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:20.691529 32742 raft_consensus.cc:2258] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:20.691715 32741 raft_consensus.cc:2258] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:20.691758 32392 tablet_service.cc:1514] Processing DeleteTablet for tablet 725eb05fa29c4803ab35f05289d93073 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:20 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:51522
I20260812 07:00:20.691956 32743 tablet_replica.cc:330] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: stopping tablet replica
I20260812 07:00:20.692080 32743 raft_consensus.cc:2229] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:20.692224 32743 raft_consensus.cc:2258] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:20.692456 32741 ts_tablet_manager.cc:1905] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:20.693090 32743 ts_tablet_manager.cc:1905] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:20.693248 31497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:20.693667 31497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:20.695216 32741 ts_tablet_manager.cc:1918] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:20.695287 32741 log.cc:1196] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/725eb05fa29c4803ab35f05289d93073
I20260812 07:00:20.695751 32741 ts_tablet_manager.cc:1939] T 725eb05fa29c4803ab35f05289d93073 P 083ba4042f0e47bab53f2b6f366085c5: Deleting consensus metadata
I20260812 07:00:20.696074 32742 ts_tablet_manager.cc:1905] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:20.697132 31474 catalog_manager.cc:4841] TS 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345): tablet 725eb05fa29c4803ab35f05289d93073 (table test-table [id=f626f5aadfc14b098adcb5e602c8c9a9]) successfully deleted
I20260812 07:00:20.698342 32743 ts_tablet_manager.cc:1918] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:20.698423 32743 log.cc:1196] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/725eb05fa29c4803ab35f05289d93073
I20260812 07:00:20.698695 32743 ts_tablet_manager.cc:1939] T 725eb05fa29c4803ab35f05289d93073 P 55fc7aa0cec14a04b4f0a13076881767: Deleting consensus metadata
I20260812 07:00:20.698787 32742 ts_tablet_manager.cc:1918] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:20.698865 32742 log.cc:1196] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/725eb05fa29c4803ab35f05289d93073
I20260812 07:00:20.699194 32742 ts_tablet_manager.cc:1939] T 725eb05fa29c4803ab35f05289d93073 P 8491ae8a46ae46f49202ea9834c31206: Deleting consensus metadata
I20260812 07:00:20.699715 31469 catalog_manager.cc:4841] TS 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829): tablet 725eb05fa29c4803ab35f05289d93073 (table test-table [id=f626f5aadfc14b098adcb5e602c8c9a9]) successfully deleted
I20260812 07:00:20.699739 32099 tablet_service.cc:1467] Processing CreateTablet for tablet 1edbfd2ae8bf4fb994bacce257947326 (DEFAULT_TABLE table=test-table [id=776a7025a12f4b1bac4ba58c4ab07a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:20.699772 31793 tablet_service.cc:1467] Processing CreateTablet for tablet 1edbfd2ae8bf4fb994bacce257947326 (DEFAULT_TABLE table=test-table [id=776a7025a12f4b1bac4ba58c4ab07a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:20.699990 32099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edbfd2ae8bf4fb994bacce257947326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:20.699990 31793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edbfd2ae8bf4fb994bacce257947326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:20.700310 32392 tablet_service.cc:1467] Processing CreateTablet for tablet 1edbfd2ae8bf4fb994bacce257947326 (DEFAULT_TABLE table=test-table [id=776a7025a12f4b1bac4ba58c4ab07a32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:20.700551 32392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1edbfd2ae8bf4fb994bacce257947326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:20.702194 32760 tablet_bootstrap.cc:492] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Bootstrap starting.
I20260812 07:00:20.702808 32763 tablet_bootstrap.cc:492] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Bootstrap starting.
I20260812 07:00:20.703233 32761 tablet_bootstrap.cc:492] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5: Bootstrap starting.
I20260812 07:00:20.703728 32763 tablet_bootstrap.cc:654] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:20.704169 32760 tablet_bootstrap.cc:654] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:20.704200 32761 tablet_bootstrap.cc:654] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:20.704327 31468 catalog_manager.cc:4841] TS 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359): tablet 725eb05fa29c4803ab35f05289d93073 (table test-table [id=f626f5aadfc14b098adcb5e602c8c9a9]) successfully deleted
I20260812 07:00:20.705418 32760 tablet_bootstrap.cc:492] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: No bootstrap required, opened a new log
I20260812 07:00:20.705418 32763 tablet_bootstrap.cc:492] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: No bootstrap required, opened a new log
I20260812 07:00:20.705509 32760 ts_tablet_manager.cc:1397] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260812 07:00:20.705523 32763 ts_tablet_manager.cc:1397] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260812 07:00:20.705938 32760 raft_consensus.cc:348] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.705938 32763 raft_consensus.cc:348] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.706069 32760 raft_consensus.cc:374] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:20.706069 32763 raft_consensus.cc:374] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:20.706115 32760 raft_consensus.cc:729] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491ae8a46ae46f49202ea9834c31206, State: Initialized, Role: FOLLOWER
I20260812 07:00:20.706115 32763 raft_consensus.cc:729] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55fc7aa0cec14a04b4f0a13076881767, State: Initialized, Role: FOLLOWER
I20260812 07:00:20.706243 32760 consensus_queue.cc:260] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.706243 32763 consensus_queue.cc:260] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.706511 32760 ts_tablet_manager.cc:1428] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260812 07:00:20.706530 32763 ts_tablet_manager.cc:1428] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:20.707566 32761 tablet_bootstrap.cc:492] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5: No bootstrap required, opened a new log
I20260812 07:00:20.707661 32761 ts_tablet_manager.cc:1397] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 07:00:20.708173 32761 raft_consensus.cc:348] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.708317 32761 raft_consensus.cc:374] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:20.708356 32761 raft_consensus.cc:729] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ba4042f0e47bab53f2b6f366085c5, State: Initialized, Role: FOLLOWER
I20260812 07:00:20.708480 32761 consensus_queue.cc:260] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.708777 32761 ts_tablet_manager.cc:1428] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:20.857471 32766 raft_consensus.cc:482] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:20.857652 32766 raft_consensus.cc:504] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.858825 32766 leader_election.cc:288] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345), 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829)
I20260812 07:00:20.863153 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.859297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:51253 (local address 127.27.110.195:32829)
0812 07:00:20.859441 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.859446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.859568 (+ 122us) server_negotiation.cc:407] Connection header received
0812 07:00:20.859764 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.859766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.859812 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.859906 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:20.860331 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.860834 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.862045 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.862460 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.862606 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.862669 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.862926 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.863001 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:00:20.863155 318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.859495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:36199 (local address 127.27.110.193:33345)
0812 07:00:20.859616 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.859620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.859636 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:20.859711 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.859714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.859770 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.859878 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:20.860199 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.860740 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.862068 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.862485 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.862626 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.862684 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.862932 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.863014 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:00:20.863178 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.859121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.27.110.194:51253)
0812 07:00:20.859476 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.859493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.859624 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.860022 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.860025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.860039 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:20.860201 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.860208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.860959 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.860965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.861904 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.861914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.862613 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.862617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.862784 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.862812 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.862905 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.862983 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":65,"thread_start_us":124,"threads_started":1}
I20260812 07:00:20.863200 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.859035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33345 (local address 127.27.110.194:36199)
0812 07:00:20.859429 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.859459 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.859616 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.859900 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.859904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.859921 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:20.860067 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.860075 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.860893 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.860897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.861916 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.861927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.862618 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.862622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.862799 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.862818 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.862913 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.862996 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":110,"thread_start_us":139,"threads_started":1}
I20260812 07:00:20.863612 32435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edbfd2ae8bf4fb994bacce257947326" candidate_uuid: "8491ae8a46ae46f49202ea9834c31206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc7aa0cec14a04b4f0a13076881767" is_pre_election: true
I20260812 07:00:20.863780 32435 raft_consensus.cc:2454] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8491ae8a46ae46f49202ea9834c31206 in term 0.
I20260812 07:00:20.863885 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edbfd2ae8bf4fb994bacce257947326" candidate_uuid: "8491ae8a46ae46f49202ea9834c31206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5" is_pre_election: true
I20260812 07:00:20.864054 31834 raft_consensus.cc:2454] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8491ae8a46ae46f49202ea9834c31206 in term 0.
I20260812 07:00:20.864215 31998 leader_election.cc:302] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55fc7aa0cec14a04b4f0a13076881767, 8491ae8a46ae46f49202ea9834c31206; no voters:
I20260812 07:00:20.864516 32766 raft_consensus.cc:2781] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:20.864574 32766 raft_consensus.cc:482] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:20.864605 32766 raft_consensus.cc:3037] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:20.865753 32766 raft_consensus.cc:504] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.866159 32766 leader_election.cc:288] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [CANDIDATE]: Term 1 election: Requested vote from peers 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345), 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829)
I20260812 07:00:20.866521 31834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edbfd2ae8bf4fb994bacce257947326" candidate_uuid: "8491ae8a46ae46f49202ea9834c31206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "083ba4042f0e47bab53f2b6f366085c5"
I20260812 07:00:20.866580 32435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1edbfd2ae8bf4fb994bacce257947326" candidate_uuid: "8491ae8a46ae46f49202ea9834c31206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55fc7aa0cec14a04b4f0a13076881767"
I20260812 07:00:20.866640 31834 raft_consensus.cc:3037] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:20.866737 32435 raft_consensus.cc:3037] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:20.867655 31834 raft_consensus.cc:2454] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8491ae8a46ae46f49202ea9834c31206 in term 1.
I20260812 07:00:20.867872 32435 raft_consensus.cc:2454] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8491ae8a46ae46f49202ea9834c31206 in term 1.
I20260812 07:00:20.868525 32001 leader_election.cc:302] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 083ba4042f0e47bab53f2b6f366085c5, 8491ae8a46ae46f49202ea9834c31206; no voters:
I20260812 07:00:20.868743 32766 raft_consensus.cc:2781] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:20.868943 32766 raft_consensus.cc:686] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 1 LEADER]: Becoming Leader. State: Replica: 8491ae8a46ae46f49202ea9834c31206, State: Running, Role: LEADER
I20260812 07:00:20.869374 32766 consensus_queue.cc:237] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } }
I20260812 07:00:20.872023 31497 catalog_manager.cc:5495] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194). New cstate: current_term: 1 leader_uuid: "8491ae8a46ae46f49202ea9834c31206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8491ae8a46ae46f49202ea9834c31206" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 40359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:20.910290 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.906827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.27.110.194:40359)
0812 07:00:20.906945 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.906949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.906971 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:00:20.907151 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.907154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.907223 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.907320 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:20.907539 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.908067 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.908937 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.909161 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.909245 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.909512 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.909970 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.910059 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.910133 (+ 74us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.910181 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:00:20.920955 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.917374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:32829 (local address 127.27.110.194:37291)
0812 07:00:20.917750 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.917767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.917866 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.918139 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.918142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.918151 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:20.918284 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.918290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.919032 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.919036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.919713 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.919723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.920188 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.920192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.920294 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.920336 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.920636 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.920646 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.920712 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.920791 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":164,"threads_started":1}
I20260812 07:00:20.920917 304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.917696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:37291 (local address 127.27.110.195:32829)
0812 07:00:20.917830 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.917835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.917849 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:20.917959 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.917962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.918030 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.918112 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:20.918397 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.918902 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.919832 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.920077 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.920185 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.920478 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.920547 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.920629 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.920722 (+ 93us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.920771 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 07:00:20.927795 32435 raft_consensus.cc:1261] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Refusing update from remote peer 8491ae8a46ae46f49202ea9834c31206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:20.927819 31834 raft_consensus.cc:1261] T 1edbfd2ae8bf4fb994bacce257947326 P 083ba4042f0e47bab53f2b6f366085c5 [term 1 FOLLOWER]: Refusing update from remote peer 8491ae8a46ae46f49202ea9834c31206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:20.928294 32766 consensus_queue.cc:1035] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55fc7aa0cec14a04b4f0a13076881767" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 32829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:20.928509 359 consensus_queue.cc:1035] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "083ba4042f0e47bab53f2b6f366085c5" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:20.947146 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.943787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:33789 (local address 127.27.110.194:40359)
0812 07:00:20.943888 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:20.943893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:20.943935 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:00:20.944087 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:20.944091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:20.944145 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:20.944235 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:20.944493 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.945027 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.945882 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.946163 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.946271 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.946686 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.946758 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.946834 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:20.946928 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 07:00:20.946980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260812 07:00:20.947176 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:20.943712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:40359 (local address 127.27.110.195:33789)
0812 07:00:20.943872 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:20.943888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:20.943985 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:20.944247 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:20.944250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:20.944258 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:20.944375 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.944382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.945130 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.945132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.945745 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:20.945755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:20.946394 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:20.946396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:20.946463 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:20.946492 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:20.946844 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:20.946853 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 07:00:20.946918 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 07:00:20.947006 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 07:00:20.948175 32708 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 07:00:22.509094 31497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:22.509238 31497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:22.511498 31497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b003ec6655e45bb821388253ec55142: Sending DeleteTablet for 3 replicas of tablet 1edbfd2ae8bf4fb994bacce257947326
I20260812 07:00:22.512028 31793 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edbfd2ae8bf4fb994bacce257947326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:22 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43876
I20260812 07:00:22.512084 32099 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edbfd2ae8bf4fb994bacce257947326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:22 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48702
I20260812 07:00:22.512197 32392 tablet_service.cc:1514] Processing DeleteTablet for tablet 1edbfd2ae8bf4fb994bacce257947326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:22 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:51522
I20260812 07:00:22.512392 426 tablet_replica.cc:330] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: stopping tablet replica
I20260812 07:00:22.512545 426 raft_consensus.cc:2229] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:22.512569 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 31654
I20260812 07:00:22.522229 426 raft_consensus.cc:2258] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:22.522572 428 tablet_replica.cc:330] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: stopping tablet replica
I20260812 07:00:22.522730 428 raft_consensus.cc:2229] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:22.522876 428 raft_consensus.cc:2258] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:22.523092 426 ts_tablet_manager.cc:1905] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:22.523648 428 ts_tablet_manager.cc:1905] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:22.525719 428 ts_tablet_manager.cc:1918] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:00:22.525774 428 log.cc:1196] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/1edbfd2ae8bf4fb994bacce257947326
I20260812 07:00:22.525972 428 ts_tablet_manager.cc:1939] T 1edbfd2ae8bf4fb994bacce257947326 P 55fc7aa0cec14a04b4f0a13076881767: Deleting consensus metadata
I20260812 07:00:22.526674 31469 catalog_manager.cc:4841] TS 55fc7aa0cec14a04b4f0a13076881767 (127.27.110.195:32829): tablet 1edbfd2ae8bf4fb994bacce257947326 (table test-table [id=776a7025a12f4b1bac4ba58c4ab07a32]) successfully deleted
I20260812 07:00:22.526894 426 ts_tablet_manager.cc:1918] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 07:00:22.526990 426 log.cc:1196] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/1edbfd2ae8bf4fb994bacce257947326
I20260812 07:00:22.527333 426 ts_tablet_manager.cc:1939] T 1edbfd2ae8bf4fb994bacce257947326 P 8491ae8a46ae46f49202ea9834c31206: Deleting consensus metadata
I20260812 07:00:22.528083 31468 catalog_manager.cc:4841] TS 8491ae8a46ae46f49202ea9834c31206 (127.27.110.194:40359): tablet 1edbfd2ae8bf4fb994bacce257947326 (table test-table [id=776a7025a12f4b1bac4ba58c4ab07a32]) successfully deleted
W20260812 07:00:22.531808 31474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:33345 (error 108)
I20260812 07:00:22.532014 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 31956
I20260812 07:00:22.541950 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:22.535073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33345 (local address 127.0.0.1:43894)
0812 07:00:22.541697 (+ 6624us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:22.541804 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:33345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6501,"thread_start_us":1340,"threads_started":1}
W20260812 07:00:22.542089 31474 catalog_manager.cc:4584] TS 083ba4042f0e47bab53f2b6f366085c5 (127.27.110.193:33345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1edbfd2ae8bf4fb994bacce257947326: Network error: Client connection negotiation failed: client connection to 127.27.110.193:33345: connect: Connection refused (error 111)
I20260812 07:00:22.552889 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 32250
I20260812 07:00:22.571805 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 31433
2026-08-12T07:00:22Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7007 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:22 dist-test-slave-w4v5 krb5kdc[436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:22 dist-test-slave-w4v5 krb5kdc[436](info): set up 2 sockets
Aug 12 07:00:22 dist-test-slave-w4v5 krb5kdc[436](info): commencing operation
krb5kdc: starting...
W20260812 07:00:23.894109 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.288s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:23 dist-test-slave-w4v5 krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:24.023680 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:37345
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:37345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:24.140367 468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:24.140692 468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:24.140776 468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:24.144866 468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:24.144950 468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:24.144987 468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:24.145038 468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:24.145087 468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:24.150542 468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:37345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:37345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:24.151707 468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:24.152740 468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:24.161293 491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.161309 483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.161382 486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:24.161610 468 server_base.cc:1034] running on GCE node
I20260812 07:00:24.162228 468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:24.163373 468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:24.164549 468 hybrid_clock.cc:648] HybridClock initialized: now 1786518024164529 us; error 62 us; skew 500 ppm
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:24.167441 468 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:24.168320 468 webserver.cc:460] Webserver started at http://127.27.110.254:39461/ using document root <none> and password file <none>
I20260812 07:00:24.168884 468 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:24.168974 468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:24.169205 468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:24.171340 468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "b18b02b8597c4c7a996252b362f67c55"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "5cd476e80d22cf641f463fe1179f2318"
server_key_iv: "05f97387bba4fa841ddd8dc6b5c4e00f"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.171909 468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "b18b02b8597c4c7a996252b362f67c55"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "5cd476e80d22cf641f463fe1179f2318"
server_key_iv: "05f97387bba4fa841ddd8dc6b5c4e00f"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.175784 468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:00:24.178407 504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.179703 468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:24.179838 468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "b18b02b8597c4c7a996252b362f67c55"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "5cd476e80d22cf641f463fe1179f2318"
server_key_iv: "05f97387bba4fa841ddd8dc6b5c4e00f"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.179946 468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:24.204557 468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:24.207577 468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:24.207785 468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:24.214959 468 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:37345
I20260812 07:00:24.215075 614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:37345 every 8 connection(s)
I20260812 07:00:24.215996 468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:24.218901 615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:24.220458 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 468
I20260812 07:00:24.220592 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:24.220850 28091 external_mini_cluster.cc:1428] Setting key 76fe5cc22708e54e356c15cb3db50932
I20260812 07:00:24.225636 615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55: Bootstrap starting.
I20260812 07:00:24.228315 615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:24.229255 615 log.cc:824] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:24.231535 615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55: No bootstrap required, opened a new log
I20260812 07:00:24.236661 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.222300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53564 (local address 127.27.110.254:37345)
0812 07:00:24.222767 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:24.222789 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:24.222835 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:00:24.223706 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:24.223730 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:24.224161 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:24.224529 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:24.225230 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.226674 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.227636 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.228334 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.228496 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.231269 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:24.231275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:24.231294 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:24.231327 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:24.233351 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.233810 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.233816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.233821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.234011 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.234301 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.234304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.234306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.234871 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:24.235111 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:24.235329 (+ 218us) server_negotiation.cc:299] Negotiation successful
0812 07:00:24.235645 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":115,"threads_started":1}
I20260812 07:00:24.236615 615 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b02b8597c4c7a996252b362f67c55" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37345 } }
I20260812 07:00:24.236892 615 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:24.236955 615 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18b02b8597c4c7a996252b362f67c55, State: Initialized, Role: FOLLOWER
I20260812 07:00:24.237421 615 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b02b8597c4c7a996252b362f67c55" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37345 } }
I20260812 07:00:24.237587 615 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:24.237665 615 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:24.237771 615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:24.238955 615 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b02b8597c4c7a996252b362f67c55" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37345 } }
I20260812 07:00:24.239353 615 leader_election.cc:302] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18b02b8597c4c7a996252b362f67c55; no voters:
I20260812 07:00:24.239681 615 leader_election.cc:288] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:24.239843 624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:24.240068 624 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [term 1 LEADER]: Becoming Leader. State: Replica: b18b02b8597c4c7a996252b362f67c55, State: Running, Role: LEADER
I20260812 07:00:24.240447 624 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b02b8597c4c7a996252b362f67c55" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37345 } }
I20260812 07:00:24.240943 615 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:24.242172 629 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18b02b8597c4c7a996252b362f67c55. Latest consensus state: current_term: 1 leader_uuid: "b18b02b8597c4c7a996252b362f67c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b02b8597c4c7a996252b362f67c55" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37345 } } }
I20260812 07:00:24.242182 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18b02b8597c4c7a996252b362f67c55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b02b8597c4c7a996252b362f67c55" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37345 } } }
I20260812 07:00:24.242313 629 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:24.242321 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:24.242743 639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:24.245779 639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:24.250571 639 catalog_manager.cc:1347] Generated new cluster ID: fef17b8af9344199a1fc94bfc0f63614
I20260812 07:00:24.250638 639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:24.264868 639 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:24.265983 639 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:24.272724 639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18b02b8597c4c7a996252b362f67c55: Generated new TSK 0
I20260812 07:00:24.273471 639 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:24.302558 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37345
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:24.418147 661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:24.418439 661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:24.418516 661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:24.422675 661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:24.422747 661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:24.422851 661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:24.428285 661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:24.429399 661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:24.430811 661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:24.440577 670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.440572 667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.440572 668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:24.440621 661 server_base.cc:1034] running on GCE node
I20260812 07:00:24.441131 661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:24.441846 661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:24.443027 661 hybrid_clock.cc:648] HybridClock initialized: now 1786518024442992 us; error 51 us; skew 500 ppm
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:24.445921 661 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:24.446852 661 webserver.cc:460] Webserver started at http://127.27.110.193:37849/ using document root <none> and password file <none>
I20260812 07:00:24.447450 661 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:24.447558 661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:24.447845 661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:24.450114 661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "745d0c1373a14ffda5e893979683e373"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "a75ced8f427406934b835e57b1ffa190"
server_key_iv: "9e683e18d727f87ff1cc759d6ca03c2a"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.450711 661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "745d0c1373a14ffda5e893979683e373"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "a75ced8f427406934b835e57b1ffa190"
server_key_iv: "9e683e18d727f87ff1cc759d6ca03c2a"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.454396 661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:00:24.457008 687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.458213 661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:00:24.458344 661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "745d0c1373a14ffda5e893979683e373"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "a75ced8f427406934b835e57b1ffa190"
server_key_iv: "9e683e18d727f87ff1cc759d6ca03c2a"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.458452 661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:24.470608 661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:24.473531 661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:24.473729 661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:24.474280 661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:24.475060 661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:24.475173 661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.475241 661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:24.475286 661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.484205 661 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:33109
I20260812 07:00:24.484336 924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:33109 every 8 connection(s)
I20260812 07:00:24.485110 661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:24.489764 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 661
I20260812 07:00:24.489905 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:24.490199 28091 external_mini_cluster.cc:1428] Setting key 8d76c7a5685e2cb961a9747d9bd58bba
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:24.499900 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.486937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:37593 (local address 127.27.110.254:37345)
0812 07:00:24.487107 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:24.487113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:24.488016 (+ 903us) server_negotiation.cc:407] Connection header received
0812 07:00:24.489532 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:24.489536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:24.489593 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:24.489697 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:24.491162 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.491791 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.492920 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.493155 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.493261 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.496078 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:24.496081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:24.496083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:24.496106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:24.497549 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.498227 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.498230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.498231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.498302 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.498722 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.498725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.498726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.498888 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:24.498960 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:24.499653 (+ 693us) server_negotiation.cc:299] Negotiation successful
0812 07:00:24.499776 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:00:24.501123 931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.487414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37345 (local address 127.27.110.193:37593)
0812 07:00:24.487867 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:24.487909 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:24.489320 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:24.489972 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:24.489986 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:24.490410 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:24.490995 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:24.491009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.491919 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.491922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:24.492796 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:24.492806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.493303 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.493307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:24.493472 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.494052 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:24.494068 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:24.495909 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:24.497675 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:24.497683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:24.497700 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:24.498113 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:24.498396 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:24.498402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:24.498404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:24.498606 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:24.498999 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:24.499006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:24.499382 (+ 376us) client_negotiation.cc:769] Sending connection context
0812 07:00:24.499606 (+ 224us) client_negotiation.cc:240] Negotiation successful
0812 07:00:24.499988 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:24.502406 925 heartbeater.cc:347] Connected to a master server at 127.27.110.254:37345
I20260812 07:00:24.502655 925 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:24.503217 925 heartbeater.cc:511] Master 127.27.110.254:37345 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:24.504760 546 ts_manager.cc:194] Registered new tserver with Master: 745d0c1373a14ffda5e893979683e373 (127.27.110.193:33109)
I20260812 07:00:24.506009 546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:37593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:24.520399 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37345
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:24.635066 936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:24.635352 936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:24.635428 936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:24.639430 936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:24.639478 936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:24.639561 936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:24.644893 936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:24.645881 936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:24.646868 936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:24.656376 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.656392 949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.656420 955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:24.656689 936 server_base.cc:1034] running on GCE node
I20260812 07:00:24.657014 936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:24.657606 936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:24.658761 936 hybrid_clock.cc:648] HybridClock initialized: now 1786518024658740 us; error 34 us; skew 500 ppm
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:24.661593 936 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:24.662438 936 webserver.cc:460] Webserver started at http://127.27.110.194:46197/ using document root <none> and password file <none>
I20260812 07:00:24.662994 936 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:24.663056 936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:24.663328 936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:24.665315 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "75602653fc654b34882ab87c02448c66"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "9b7b56bbd03d2342fb99ef93a1ac747a"
server_key_iv: "6f47f8f684fd7bb954ef63d2a6aacaa3"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.665879 936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "75602653fc654b34882ab87c02448c66"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "9b7b56bbd03d2342fb99ef93a1ac747a"
server_key_iv: "6f47f8f684fd7bb954ef63d2a6aacaa3"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.669674 936 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260812 07:00:24.672381 970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.673529 936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:24.673655 936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "75602653fc654b34882ab87c02448c66"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "9b7b56bbd03d2342fb99ef93a1ac747a"
server_key_iv: "6f47f8f684fd7bb954ef63d2a6aacaa3"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.673769 936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:24.697536 936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:24.700559 936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:24.700774 936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:24.701404 936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:24.702292 936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:24.702351 936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.702422 936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:24.702458 936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.711846 936 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:38803
I20260812 07:00:24.711936 1223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:38803 every 8 connection(s)
I20260812 07:00:24.712805 936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:24.717005 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 936
I20260812 07:00:24.717141 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:24.717429 28091 external_mini_cluster.cc:1428] Setting key b1517c91fa170968d1b3c5b98b865e50
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:24.728436 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.714628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:44515 (local address 127.27.110.254:37345)
0812 07:00:24.714767 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:24.714771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:24.715681 (+ 910us) server_negotiation.cc:407] Connection header received
0812 07:00:24.717051 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:24.717055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:24.717112 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:24.717198 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:24.718665 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.719470 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.720649 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.721039 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.721170 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.724494 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:24.724497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:24.724499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:24.724524 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:24.726035 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.726735 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.726738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.726740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.726820 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.727253 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.727255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.727256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.727438 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:24.727519 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:24.728207 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 07:00:24.728307 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:24.729718 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.715019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37345 (local address 127.27.110.194:44515)
0812 07:00:24.715520 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:24.715563 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:24.716834 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:24.717375 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:24.717381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:24.717843 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:24.718514 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:24.718533 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.719614 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.719617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:24.720520 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:24.720530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.721233 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.721239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:24.721473 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.722167 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:24.722193 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:24.724302 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:24.726175 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:24.726185 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:24.726206 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:24.726617 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:24.726916 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:24.726922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:24.726925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:24.727159 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:24.727556 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:24.727565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:24.727930 (+ 365us) client_negotiation.cc:769] Sending connection context
0812 07:00:24.728164 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 07:00:24.728543 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":117,"threads_started":1}
I20260812 07:00:24.730957 1225 heartbeater.cc:347] Connected to a master server at 127.27.110.254:37345
I20260812 07:00:24.731238 1225 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:24.731851 1225 heartbeater.cc:511] Master 127.27.110.254:37345 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:00:24.732936 546 ts_manager.cc:194] Registered new tserver with Master: 75602653fc654b34882ab87c02448c66 (127.27.110.194:38803)
I20260812 07:00:24.733507 546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:44515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:24.749553 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37345
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:24.868062 1236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:24.868333 1236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:24.868376 1236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:24.872375 1236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:24.872435 1236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:24.872572 1236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:24.878095 1236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:24.879261 1236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:24.880302 1236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:24.890277 1243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.890372 1244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:24.890359 1247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:24.890558 1236 server_base.cc:1034] running on GCE node
I20260812 07:00:24.891053 1236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:24.891757 1236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:24.892936 1236 hybrid_clock.cc:648] HybridClock initialized: now 1786518024892906 us; error 45 us; skew 500 ppm
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:24.895783 1236 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:24.896687 1236 webserver.cc:460] Webserver started at http://127.27.110.195:34991/ using document root <none> and password file <none>
I20260812 07:00:24.897265 1236 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:24.897357 1236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:24.897586 1236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:24.899718 1236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "8cf24d51c28c45289f487c54249d3961"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "cc70c00aa4b07984cae0a8467abec520"
server_key_iv: "9d92cd5acc8814d3d05bdc422040cbad"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.900310 1236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "8cf24d51c28c45289f487c54249d3961"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "cc70c00aa4b07984cae0a8467abec520"
server_key_iv: "9d92cd5acc8814d3d05bdc422040cbad"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.904119 1236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:00:24.906649 1255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.907900 1236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:24.908038 1236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "8cf24d51c28c45289f487c54249d3961"
format_stamp: "Formatted at 2026-08-12 07:00:24 on dist-test-slave-w4v5"
server_key: "cc70c00aa4b07984cae0a8467abec520"
server_key_iv: "9d92cd5acc8814d3d05bdc422040cbad"
server_key_version: "encryptionkey@0"
I20260812 07:00:24.908151 1236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:24.931131 1236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:24.934329 1236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:24.934554 1236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:24.935217 1236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:24.936096 1236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:24.936167 1236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.936236 1236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:24.936282 1236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:24.946201 1236 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:35001
I20260812 07:00:24.946225 1497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:35001 every 8 connection(s)
I20260812 07:00:24.947126 1236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
Aug 12 07:00:24 dist-test-slave-w4v5 krb5kdc[436](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518024, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:24.956897 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1236
I20260812 07:00:24.957038 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:24.957327 28091 external_mini_cluster.cc:1428] Setting key e65aea208e9a53aee0ca826c5094ef0a
I20260812 07:00:24.961212 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.948998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:47321 (local address 127.27.110.254:37345)
0812 07:00:24.949152 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:24.949157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:24.950047 (+ 890us) server_negotiation.cc:407] Connection header received
0812 07:00:24.951399 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:24.951402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:24.951449 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:24.951525 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:24.952685 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.953222 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.954500 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.954731 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.954816 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.957530 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:24.957533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:24.957535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:24.957559 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:24.959236 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.959808 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.959811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.959813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.959890 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:24.960209 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:24.960213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:24.960214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:24.960442 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:24.960573 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:24.960976 (+ 403us) server_negotiation.cc:299] Negotiation successful
0812 07:00:24.961075 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:00:24.962110 1500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:24.949428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37345 (local address 127.27.110.195:47321)
0812 07:00:24.949884 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:24.949931 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:24.951191 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:24.951659 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:24.951666 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:24.952034 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:24.952518 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:24.952531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.953340 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.953342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:24.954366 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:24.954378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:24.954878 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:24.954881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:24.955022 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:24.955673 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:24.955689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:24.957349 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:24.959372 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:24.959377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:24.959390 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:24.959719 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:24.959971 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:24.959974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:24.959975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:24.960112 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:24.960540 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:24.960545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:24.960771 (+ 226us) client_negotiation.cc:769] Sending connection context
0812 07:00:24.960927 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 07:00:24.961270 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":106,"threads_started":1}
I20260812 07:00:24.963265 1498 heartbeater.cc:347] Connected to a master server at 127.27.110.254:37345
I20260812 07:00:24.963475 1498 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:24.963919 1498 heartbeater.cc:511] Master 127.27.110.254:37345 requested a full tablet report, sending...
I20260812 07:00:24.964845 546 ts_manager.cc:194] Registered new tserver with Master: 8cf24d51c28c45289f487c54249d3961 (127.27.110.195:35001)
I20260812 07:00:24.965358 546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:47321
I20260812 07:00:24.971153 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 07:00:24.986402 621 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:24.984947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.27.110.254:37345)
0812 07:00:24.985106 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:24.985111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:24.985265 (+ 154us) server_negotiation.cc:407] Connection header received
0812 07:00:24.985380 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:24.985383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:24.985428 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:24.985502 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:24.986256 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53566: BlockingRecv error: recv got EOF from 127.0.0.1:53566 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:00:24.987486 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 661
I20260812 07:00:25.004199 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 936
I20260812 07:00:25.019956 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1236
I20260812 07:00:25.037024 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 468
2026-08-12T07:00:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2462 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:25 dist-test-slave-w4v5 krb5kdc[1517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:25 dist-test-slave-w4v5 krb5kdc[1517](info): set up 2 sockets
Aug 12 07:00:25 dist-test-slave-w4v5 krb5kdc[1517](info): commencing operation
krb5kdc: starting...
W20260812 07:00:26.360458 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:26 dist-test-slave-w4v5 krb5kdc[1517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:26.483600 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:35099
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:35099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:26.596509 1545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:26.596784 1545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:26.596824 1545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:26.600840 1545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:26.600898 1545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:26.600917 1545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:26.600934 1545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:26.600948 1545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:26.606050 1545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:35099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:35099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:26.607043 1545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:26.608100 1545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:26.616652 1554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:26.616833 1545 server_base.cc:1034] running on GCE node
W20260812 07:00:26.616699 1564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:26.616938 1561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:26.617558 1545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:26.618692 1545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:26.619874 1545 hybrid_clock.cc:648] HybridClock initialized: now 1786518026619859 us; error 26 us; skew 500 ppm
Aug 12 07:00:26 dist-test-slave-w4v5 krb5kdc[1517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518026, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:26.622702 1545 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:26.623652 1545 webserver.cc:460] Webserver started at http://127.27.110.254:38581/ using document root <none> and password file <none>
I20260812 07:00:26.624182 1545 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:26.624243 1545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:26.624409 1545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:26.626407 1545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "47578d4c773f4048a564714eaf583665"
format_stamp: "Formatted at 2026-08-12 07:00:26 on dist-test-slave-w4v5"
server_key: "071bcbbf9077105ed19fa6e15ff494c1"
server_key_iv: "465d5e99fd6a264dc65d30e2dca81767"
server_key_version: "encryptionkey@0"
I20260812 07:00:26.626924 1545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "47578d4c773f4048a564714eaf583665"
format_stamp: "Formatted at 2026-08-12 07:00:26 on dist-test-slave-w4v5"
server_key: "071bcbbf9077105ed19fa6e15ff494c1"
server_key_iv: "465d5e99fd6a264dc65d30e2dca81767"
server_key_version: "encryptionkey@0"
I20260812 07:00:26.630605 1545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:26.633908 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:26.635351 1545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:26.635483 1545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "47578d4c773f4048a564714eaf583665"
format_stamp: "Formatted at 2026-08-12 07:00:26 on dist-test-slave-w4v5"
server_key: "071bcbbf9077105ed19fa6e15ff494c1"
server_key_iv: "465d5e99fd6a264dc65d30e2dca81767"
server_key_version: "encryptionkey@0"
I20260812 07:00:26.635601 1545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:26.651618 1545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:26.654814 1545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:26.655015 1545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:26.662292 1545 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:35099
I20260812 07:00:26.662634 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:35099 every 8 connection(s)
I20260812 07:00:26.663275 1545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:26.666002 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:26.669457 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1545
I20260812 07:00:26.669615 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:26.669871 28091 external_mini_cluster.cc:1428] Setting key 2d31e195ba5d3a74fbb58ccb75debeeb
I20260812 07:00:26.671716 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665: Bootstrap starting.
I20260812 07:00:26.673863 1713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:26.674588 1713 log.cc:824] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:26.676600 1713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665: No bootstrap required, opened a new log
Aug 12 07:00:26 dist-test-slave-w4v5 krb5kdc[1517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:26.680603 1713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47578d4c773f4048a564714eaf583665" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35099 } }
I20260812 07:00:26.680905 1713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:26.680967 1713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47578d4c773f4048a564714eaf583665, State: Initialized, Role: FOLLOWER
I20260812 07:00:26.681448 1713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47578d4c773f4048a564714eaf583665" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35099 } }
I20260812 07:00:26.681586 1713 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:26.681674 1713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:26.681775 1713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:26.682878 1713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47578d4c773f4048a564714eaf583665" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35099 } }
I20260812 07:00:26.683243 1713 leader_election.cc:302] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47578d4c773f4048a564714eaf583665; no voters:
I20260812 07:00:26.683679 1713 leader_election.cc:288] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:26.683740 1722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:26.683928 1722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [term 1 LEADER]: Becoming Leader. State: Replica: 47578d4c773f4048a564714eaf583665, State: Running, Role: LEADER
I20260812 07:00:26.684290 1722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47578d4c773f4048a564714eaf583665" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35099 } }
I20260812 07:00:26.684513 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:26.671237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37802 (local address 127.27.110.254:35099)
0812 07:00:26.671699 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:26.671715 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:26.671749 (+ 34us) server_negotiation.cc:407] Connection header received
0812 07:00:26.672405 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:26.672422 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:26.672751 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:26.673083 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:26.673927 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:26.674996 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:26.675931 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:26.676599 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:26.676780 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:26.679256 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:26.679262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:26.679278 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:26.679311 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:26.681305 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:26.681804 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:26.681810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:26.681815 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:26.681961 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:26.682224 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:26.682227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:26.682229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:26.682754 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:26.682953 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:26.683255 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 07:00:26.683541 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":147,"threads_started":1}
I20260812 07:00:26.684923 1713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:26.686144 1723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47578d4c773f4048a564714eaf583665. Latest consensus state: current_term: 1 leader_uuid: "47578d4c773f4048a564714eaf583665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47578d4c773f4048a564714eaf583665" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35099 } } }
I20260812 07:00:26.686266 1723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:26.687569 1724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47578d4c773f4048a564714eaf583665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47578d4c773f4048a564714eaf583665" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 35099 } } }
I20260812 07:00:26.687826 1724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:26.687927 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260812 07:00:26.689684 1757 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:26.689775 1757 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:26.689884 1742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:26.690572 1742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:26.695632 1742 catalog_manager.cc:1347] Generated new cluster ID: 13d4367ff26e430598bf50b68ee45ffc
I20260812 07:00:26.695699 1742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:26.717506 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:26.710784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.27.110.254:35099)
0812 07:00:26.710909 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:26.710915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:26.711224 (+ 309us) server_negotiation.cc:407] Connection header received
0812 07:00:26.711343 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:26.711349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:26.711412 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:26.711506 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:26.712086 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:26.712930 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:26.714081 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:26.714447 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:26.714555 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:26.715522 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:26.715527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:26.715530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:26.716860 (+ 1330us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:26.716901 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:26.716923 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:26.717021 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:26.717126 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 07:00:26.717198 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:00:26.721699 1742 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:26.722653 1742 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:26.728139 1742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47578d4c773f4048a564714eaf583665: Generated new TSK 0
I20260812 07:00:26.728910 1742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:00:26.739055 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1545
2026-08-12T07:00:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1701 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:26 dist-test-slave-w4v5 krb5kdc[1780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:26 dist-test-slave-w4v5 krb5kdc[1780](info): set up 2 sockets
Aug 12 07:00:26 dist-test-slave-w4v5 krb5kdc[1780](info): commencing operation
krb5kdc: starting...
W20260812 07:00:28.053892 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:28.182900 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:40485
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:40485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:28.299126 1808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:28.299417 1808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:28.299513 1808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:28.303512 1808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:28.303586 1808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:28.303632 1808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:28.303691 1808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:28.303731 1808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:28.308969 1808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:40485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:40485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:28.309966 1808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:28.311020 1808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:28.319998 1816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:28.319993 1815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:28.319990 1818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:28.320310 1808 server_base.cc:1034] running on GCE node
I20260812 07:00:28.320907 1808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:28.321954 1808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:28.323151 1808 hybrid_clock.cc:648] HybridClock initialized: now 1786518028323133 us; error 29 us; skew 500 ppm
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:28.329216 1808 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:28.330123 1808 webserver.cc:460] Webserver started at http://127.27.110.254:45307/ using document root <none> and password file <none>
I20260812 07:00:28.330685 1808 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:28.330801 1808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:28.331043 1808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:28.333081 1808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "f4287755fe1440f3bf4886c61c97ad3f"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "41b2689af093a835cb541d04ffae0a58"
server_key_iv: "9a91b3eee298f5e30cc5edcb005053a6"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.333647 1808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "f4287755fe1440f3bf4886c61c97ad3f"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "41b2689af093a835cb541d04ffae0a58"
server_key_iv: "9a91b3eee298f5e30cc5edcb005053a6"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.337332 1808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 07:00:28.339707 1832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.340899 1808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:28.341040 1808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "f4287755fe1440f3bf4886c61c97ad3f"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "41b2689af093a835cb541d04ffae0a58"
server_key_iv: "9a91b3eee298f5e30cc5edcb005053a6"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.341150 1808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:28.363587 1808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:28.366577 1808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:28.366789 1808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:28.373956 1808 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:40485
I20260812 07:00:28.374249 1942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:40485 every 8 connection(s)
I20260812 07:00:28.374953 1808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:28.377772 1943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:28.379896 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1808
I20260812 07:00:28.380026 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:28.380329 28091 external_mini_cluster.cc:1428] Setting key 6b9842b0dab9821fe17e372ed5842072
I20260812 07:00:28.384688 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f: Bootstrap starting.
I20260812 07:00:28.387225 1943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:28.388168 1943 log.cc:824] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:28.390254 1943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f: No bootstrap required, opened a new log
I20260812 07:00:28.395030 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:28.381712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.27.110.254:40485)
0812 07:00:28.382138 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:28.382153 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:28.382195 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:00:28.382827 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:28.382843 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:28.383226 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:28.383502 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:28.384419 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.385442 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.386644 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.387176 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.387361 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:28.389960 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:28.389966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:28.389982 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:28.390012 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:28.391943 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:28.392379 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:28.392386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:28.392392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:28.392545 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:28.392817 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:28.392821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:28.392823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:28.393349 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:28.393522 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:28.393795 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 07:00:28.394081 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20260812 07:00:28.394992 1943 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4287755fe1440f3bf4886c61c97ad3f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 40485 } }
I20260812 07:00:28.395294 1943 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:28.395357 1943 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4287755fe1440f3bf4886c61c97ad3f, State: Initialized, Role: FOLLOWER
I20260812 07:00:28.395795 1943 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4287755fe1440f3bf4886c61c97ad3f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 40485 } }
I20260812 07:00:28.395910 1943 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:28.395948 1943 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:28.396052 1943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:28.397174 1943 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4287755fe1440f3bf4886c61c97ad3f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 40485 } }
I20260812 07:00:28.397513 1943 leader_election.cc:302] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4287755fe1440f3bf4886c61c97ad3f; no voters:
I20260812 07:00:28.397867 1943 leader_election.cc:288] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:28.397969 1951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:28.398200 1951 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [term 1 LEADER]: Becoming Leader. State: Replica: f4287755fe1440f3bf4886c61c97ad3f, State: Running, Role: LEADER
I20260812 07:00:28.398512 1951 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4287755fe1440f3bf4886c61c97ad3f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 40485 } }
I20260812 07:00:28.399014 1943 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:28.400274 1953 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4287755fe1440f3bf4886c61c97ad3f. Latest consensus state: current_term: 1 leader_uuid: "f4287755fe1440f3bf4886c61c97ad3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4287755fe1440f3bf4886c61c97ad3f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 40485 } } }
I20260812 07:00:28.400421 1953 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:28.400280 1952 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4287755fe1440f3bf4886c61c97ad3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4287755fe1440f3bf4886c61c97ad3f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 40485 } } }
I20260812 07:00:28.400537 1952 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:28.401360 1965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:28.403999 1965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:28.409021 1965 catalog_manager.cc:1347] Generated new cluster ID: e91c1e97a38e41889771d81ac0eee0c0
I20260812 07:00:28.409086 1965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:28.422991 1965 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:28.423897 1965 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:28.438020 1965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4287755fe1440f3bf4886c61c97ad3f: Generated new TSK 0
I20260812 07:00:28.438712 1965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:28.470268 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:40485
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:28.585421 1989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:28.585672 1989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:28.585713 1989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:28.589710 1989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:28.589766 1989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:28.589821 1989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:28.594841 1989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:28.595806 1989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:28.596804 1989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:28.606225 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:28.606297 1989 server_base.cc:1034] running on GCE node
W20260812 07:00:28.606348 1998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:28.606236 1997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:28.606935 1989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:28.607781 1989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:28.608973 1989 hybrid_clock.cc:648] HybridClock initialized: now 1786518028608948 us; error 33 us; skew 500 ppm
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:28.612270 1989 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:28.613179 1989 webserver.cc:460] Webserver started at http://127.27.110.193:38705/ using document root <none> and password file <none>
I20260812 07:00:28.613770 1989 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:28.613837 1989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:28.614086 1989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:28.616370 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "4488a60e16c24f049a20909c955bde88"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "8d39fd23c641db583c0c820043327ed9"
server_key_iv: "457a296a113fd0fcf91bff93e06804ad"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.617020 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "4488a60e16c24f049a20909c955bde88"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "8d39fd23c641db583c0c820043327ed9"
server_key_iv: "457a296a113fd0fcf91bff93e06804ad"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.621163 1989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:28.623765 2016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.625001 1989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:28.625157 1989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "4488a60e16c24f049a20909c955bde88"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "8d39fd23c641db583c0c820043327ed9"
server_key_iv: "457a296a113fd0fcf91bff93e06804ad"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.625277 1989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:28.636116 1989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:28.638877 1989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:28.639144 1989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:28.641376 1989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:28.642181 1989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:28.642252 1989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.642315 1989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:28.642359 1989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.651407 1989 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:37229
I20260812 07:00:28.651439 2262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:37229 every 8 connection(s)
I20260812 07:00:28.652271 1989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:28.657117 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1989
I20260812 07:00:28.657249 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:28.657522 28091 external_mini_cluster.cc:1428] Setting key a713d709ec6bf1721626a82a691854f3
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:28.666522 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:28.653891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:48925 (local address 127.27.110.254:40485)
0812 07:00:28.654017 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:28.654021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:28.654813 (+ 792us) server_negotiation.cc:407] Connection header received
0812 07:00:28.656271 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:28.656275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:28.656334 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:28.656432 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:28.657670 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.658370 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.659601 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.659848 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.659960 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:28.662716 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:28.662720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:28.662722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:28.662745 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:28.664217 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:28.664875 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:28.664878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:28.664879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:28.664954 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:28.665364 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:28.665366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:28.665367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:28.665529 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:28.665606 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:28.666297 (+ 691us) server_negotiation.cc:299] Negotiation successful
0812 07:00:28.666402 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 07:00:28.667649 2266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:28.654217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:40485 (local address 127.27.110.193:48925)
0812 07:00:28.654660 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:28.654705 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:28.656060 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:28.656598 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:28.656604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:28.656952 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:28.657494 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:28.657508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.658504 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.658508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:28.659475 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:28.659485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.659998 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.660002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:28.660165 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:28.660737 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:28.660753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:28.662546 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:28.664363 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:28.664372 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:28.664388 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:28.664770 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:28.665047 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:28.665054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:28.665057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:28.665265 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:28.665635 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:28.665645 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:28.666009 (+ 364us) client_negotiation.cc:769] Sending connection context
0812 07:00:28.666232 (+ 223us) client_negotiation.cc:240] Negotiation successful
0812 07:00:28.666572 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:28.668852 2263 heartbeater.cc:347] Connected to a master server at 127.27.110.254:40485
I20260812 07:00:28.669083 2263 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:28.669569 2263 heartbeater.cc:511] Master 127.27.110.254:40485 requested a full tablet report, sending...
I20260812 07:00:28.671028 1871 ts_manager.cc:194] Registered new tserver with Master: 4488a60e16c24f049a20909c955bde88 (127.27.110.193:37229)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:28.672259 1871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:48925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:28.686142 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:40485
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:28.804144 2270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:28.804389 2270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:28.804471 2270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:28.808446 2270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:28.808501 2270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:28.808610 2270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:28.813799 2270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:28.814819 2270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:28.815863 2270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:28.825568 2280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:28.825573 2283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:28.825663 2270 server_base.cc:1034] running on GCE node
W20260812 07:00:28.825573 2286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:28.826280 2270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:28.826936 2270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:28.828107 2270 hybrid_clock.cc:648] HybridClock initialized: now 1786518028828096 us; error 19 us; skew 500 ppm
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:28.830956 2270 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:28.831875 2270 webserver.cc:460] Webserver started at http://127.27.110.194:37455/ using document root <none> and password file <none>
I20260812 07:00:28.832381 2270 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:28.832441 2270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:28.832604 2270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:28.834628 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "af279142002247099ca00b413335dc15"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "0df4f1b82db147fc896857be9a0dffdc"
server_key_iv: "428142790a3d01289a02033a3ece139a"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.835213 2270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "af279142002247099ca00b413335dc15"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "0df4f1b82db147fc896857be9a0dffdc"
server_key_iv: "428142790a3d01289a02033a3ece139a"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.838761 2270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:00:28.841205 2297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.842445 2270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:00:28.842602 2270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "af279142002247099ca00b413335dc15"
format_stamp: "Formatted at 2026-08-12 07:00:28 on dist-test-slave-w4v5"
server_key: "0df4f1b82db147fc896857be9a0dffdc"
server_key_iv: "428142790a3d01289a02033a3ece139a"
server_key_version: "encryptionkey@0"
I20260812 07:00:28.842708 2270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:28.856305 2270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:28.859133 2270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:28.859331 2270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:28.859905 2270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:28.860714 2270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:28.860760 2270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.860824 2270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:28.860858 2270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:28.869760 2270 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:44463
I20260812 07:00:28.869787 2520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:44463 every 8 connection(s)
I20260812 07:00:28.870672 2270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:28.872645 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2270
I20260812 07:00:28.872808 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:28.873078 28091 external_mini_cluster.cc:1428] Setting key 27dedb92079b6dd6a3427d94b027d5f6
Aug 12 07:00:28 dist-test-slave-w4v5 krb5kdc[1780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518028, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:28.885746 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:28.872511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:51499 (local address 127.27.110.254:40485)
0812 07:00:28.872665 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:28.872670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:28.873559 (+ 889us) server_negotiation.cc:407] Connection header received
0812 07:00:28.875215 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:28.875221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:28.875284 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:28.875380 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:28.876507 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.877154 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.878673 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.879109 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.879241 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:28.881687 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:28.881702 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:28.881708 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:28.881749 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:28.883538 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:28.884076 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:28.884082 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:28.884085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:28.884185 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:28.884573 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:28.884578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:28.884580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:28.884855 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:28.884967 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:28.885460 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 07:00:28.885605 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:00:28.886690 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:28.872934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:40485 (local address 127.27.110.194:51499)
0812 07:00:28.873390 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:28.873443 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:28.874952 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:28.875523 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:28.875535 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:28.875874 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:28.876363 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:28.876377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.877292 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.877297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:28.878544 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:28.878554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:28.879249 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:28.879252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:28.879395 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:28.880066 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:28.880083 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:28.881511 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:28.883677 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:28.883682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:28.883693 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:28.883959 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:28.884313 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:28.884317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:28.884318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:28.884460 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:28.885006 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:28.885012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:28.885239 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 07:00:28.885406 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 07:00:28.885772 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:00:28.887885 2523 heartbeater.cc:347] Connected to a master server at 127.27.110.254:40485
I20260812 07:00:28.888136 2523 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:28.888645 2523 heartbeater.cc:511] Master 127.27.110.254:40485 requested a full tablet report, sending...
I20260812 07:00:28.889722 1871 ts_manager.cc:194] Registered new tserver with Master: af279142002247099ca00b413335dc15 (127.27.110.194:44463)
I20260812 07:00:28.890308 1871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:51499
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:28.903884 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:40485
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:29.017818 2534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:29.018078 2534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:29.018121 2534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:29.022337 2534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:29.022394 2534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:29.022490 2534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:29.027726 2534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:40485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:29.028842 2534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:29.029954 2534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:29.040021 2542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:29.040011 2549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:29.040122 2534 server_base.cc:1034] running on GCE node
W20260812 07:00:29.040011 2552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:29.040714 2534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:29.041378 2534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:29.042577 2534 hybrid_clock.cc:648] HybridClock initialized: now 1786518029042540 us; error 51 us; skew 500 ppm
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:29.045789 2534 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:29.046795 2534 webserver.cc:460] Webserver started at http://127.27.110.195:32885/ using document root <none> and password file <none>
I20260812 07:00:29.047434 2534 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:29.047500 2534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:29.047724 2534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:29.049811 2534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "a4a451773d55484f96ba351cf4d277ce"
format_stamp: "Formatted at 2026-08-12 07:00:29 on dist-test-slave-w4v5"
server_key: "2c30992150e3346bdc8898b56c07b2eb"
server_key_iv: "b5472bca23cb3cee1a2d3512cd979398"
server_key_version: "encryptionkey@0"
I20260812 07:00:29.050395 2534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "a4a451773d55484f96ba351cf4d277ce"
format_stamp: "Formatted at 2026-08-12 07:00:29 on dist-test-slave-w4v5"
server_key: "2c30992150e3346bdc8898b56c07b2eb"
server_key_iv: "b5472bca23cb3cee1a2d3512cd979398"
server_key_version: "encryptionkey@0"
I20260812 07:00:29.054426 2534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:00:29.057049 2564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:29.058358 2534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:29.058480 2534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "a4a451773d55484f96ba351cf4d277ce"
format_stamp: "Formatted at 2026-08-12 07:00:29 on dist-test-slave-w4v5"
server_key: "2c30992150e3346bdc8898b56c07b2eb"
server_key_iv: "b5472bca23cb3cee1a2d3512cd979398"
server_key_version: "encryptionkey@0"
I20260812 07:00:29.058585 2534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:29.071679 2534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:29.074690 2534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:29.074895 2534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:29.075554 2534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:29.076375 2534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:29.076421 2534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:29.076495 2534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:29.076535 2534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:29.085968 2534 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:36593
I20260812 07:00:29.086004 2804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:36593 every 8 connection(s)
I20260812 07:00:29.086872 2534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:29.090704 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2534
I20260812 07:00:29.090840 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:29.091209 28091 external_mini_cluster.cc:1428] Setting key 061ab30b7ac91e41f6a2b29f462d98c1
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[1780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518029, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:29.102205 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:29.088716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:36773 (local address 127.27.110.254:40485)
0812 07:00:29.088888 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:29.088893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:29.089719 (+ 826us) server_negotiation.cc:407] Connection header received
0812 07:00:29.091437 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:29.091440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:29.091536 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:29.091638 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:29.093142 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:29.093808 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:29.095020 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:29.095347 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:29.095440 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:29.098344 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:29.098349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:29.098351 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:29.098383 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:29.100224 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:29.100840 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:29.100843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:29.100844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:29.100927 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:29.101218 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:29.101221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:29.101223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:29.101403 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:29.101481 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:29.101973 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 07:00:29.102081 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:00:29.103138 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:29.089096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:40485 (local address 127.27.110.195:36773)
0812 07:00:29.089565 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:29.089609 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:29.091213 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:29.091840 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:29.091849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:29.092400 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:29.092991 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:29.093005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:29.093929 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:29.093932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:29.094842 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:29.094852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:29.095488 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:29.095491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:29.095668 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:29.096300 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:29.096317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:29.098151 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:29.100352 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:29.100363 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:29.100404 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:29.100741 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:29.100992 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:29.100994 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:29.100995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:29.101121 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:29.101493 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:29.101498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:29.101758 (+ 260us) client_negotiation.cc:769] Sending connection context
0812 07:00:29.101917 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 07:00:29.102167 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260812 07:00:29.104363 2807 heartbeater.cc:347] Connected to a master server at 127.27.110.254:40485
I20260812 07:00:29.104673 2807 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:29.105216 2807 heartbeater.cc:511] Master 127.27.110.254:40485 requested a full tablet report, sending...
I20260812 07:00:29.106300 1871 ts_manager.cc:194] Registered new tserver with Master: a4a451773d55484f96ba351cf4d277ce (127.27.110.195:36593)
I20260812 07:00:29.106890 1871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:36773
I20260812 07:00:29.116503 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[1780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[1780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518029, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:29.144124 1949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:29.135334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.27.110.254:40485)
0812 07:00:29.135513 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:29.135518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:29.135665 (+ 147us) server_negotiation.cc:407] Connection header received
0812 07:00:29.135818 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:29.135826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:29.135870 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:29.135958 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:29.136709 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:29.137232 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:29.138073 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:29.138297 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:29.138407 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:29.140969 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:29.140973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:29.140975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:29.141000 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:29.142349 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:29.142796 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:29.142804 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:29.142806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:29.142883 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:29.143206 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:29.143209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:29.143211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:29.143441 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:29.143615 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:29.143832 (+ 217us) server_negotiation.cc:299] Negotiation successful
0812 07:00:29.143958 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 07:00:29.144534 1871 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47002
I20260812 07:00:29.145565 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1989
I20260812 07:00:29.163976 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2270
I20260812 07:00:29.178049 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2534
I20260812 07:00:29.189795 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1808
2026-08-12T07:00:29Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2448 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[2826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[2826](info): set up 2 sockets
Aug 12 07:00:29 dist-test-slave-w4v5 krb5kdc[2826](info): commencing operation
krb5kdc: starting...
W20260812 07:00:30.504581 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:30 dist-test-slave-w4v5 krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:30.638484 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:46571
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:46571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:30.753405 2853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:30.753729 2853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:30.753813 2853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:30.758721 2853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:30.758809 2853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:30.758844 2853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:30.758898 2853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:30.758930 2853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:30.765527 2853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:46571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:46571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:30.766623 2853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:30.767795 2853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:30.776546 2858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:30.776605 2859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:30.776559 2863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:30.777082 2853 server_base.cc:1034] running on GCE node
I20260812 07:00:30.777781 2853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:30.778990 2853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:30.780203 2853 hybrid_clock.cc:648] HybridClock initialized: now 1786518030780182 us; error 31 us; skew 500 ppm
Aug 12 07:00:30 dist-test-slave-w4v5 krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518030, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:30.783017 2853 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:30.784005 2853 webserver.cc:460] Webserver started at http://127.27.110.254:39883/ using document root <none> and password file <none>
I20260812 07:00:30.784585 2853 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:30.784677 2853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:30.784919 2853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:30.787014 2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "8f6c73b6197045bc82635f2036cdfcc7"
format_stamp: "Formatted at 2026-08-12 07:00:30 on dist-test-slave-w4v5"
server_key: "d54ab2d2b40418996be5ba874562ef2b"
server_key_iv: "9ab52ae1d0ee353fe846ded90577e4a3"
server_key_version: "encryptionkey@0"
I20260812 07:00:30.787658 2853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "8f6c73b6197045bc82635f2036cdfcc7"
format_stamp: "Formatted at 2026-08-12 07:00:30 on dist-test-slave-w4v5"
server_key: "d54ab2d2b40418996be5ba874562ef2b"
server_key_iv: "9ab52ae1d0ee353fe846ded90577e4a3"
server_key_version: "encryptionkey@0"
I20260812 07:00:30.791504 2853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:00:30.794023 2876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:30.795334 2853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:30.795471 2853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "8f6c73b6197045bc82635f2036cdfcc7"
format_stamp: "Formatted at 2026-08-12 07:00:30 on dist-test-slave-w4v5"
server_key: "d54ab2d2b40418996be5ba874562ef2b"
server_key_iv: "9ab52ae1d0ee353fe846ded90577e4a3"
server_key_version: "encryptionkey@0"
I20260812 07:00:30.795593 2853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:30.806382 2853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:30.809265 2853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:30.809477 2853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:30.817319 2853 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:46571
I20260812 07:00:30.817350 2995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:46571 every 8 connection(s)
I20260812 07:00:30.818325 2853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:30.821295 2998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:30.825290 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2853
I20260812 07:00:30.825433 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:30.825735 28091 external_mini_cluster.cc:1428] Setting key ff6098f89e2e32b341cf90ad6f48c501
I20260812 07:00:30.827356 2998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7: Bootstrap starting.
I20260812 07:00:30.829670 2998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:30.830466 2998 log.cc:824] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:30.832423 2998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7: No bootstrap required, opened a new log
Aug 12 07:00:30 dist-test-slave-w4v5 krb5kdc[2826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:30.836813 2998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6c73b6197045bc82635f2036cdfcc7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46571 } }
I20260812 07:00:30.837046 2998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:30.837081 2998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6c73b6197045bc82635f2036cdfcc7, State: Initialized, Role: FOLLOWER
I20260812 07:00:30.837576 2998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6c73b6197045bc82635f2036cdfcc7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46571 } }
I20260812 07:00:30.837703 2998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:30.837788 2998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:30.837898 2998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:30.839047 2998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6c73b6197045bc82635f2036cdfcc7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46571 } }
I20260812 07:00:30.839454 2998 leader_election.cc:302] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f6c73b6197045bc82635f2036cdfcc7; no voters:
I20260812 07:00:30.839802 2998 leader_election.cc:288] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:30.839948 3006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:30.840157 3006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [term 1 LEADER]: Becoming Leader. State: Replica: 8f6c73b6197045bc82635f2036cdfcc7, State: Running, Role: LEADER
I20260812 07:00:30.840456 3006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6c73b6197045bc82635f2036cdfcc7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46571 } }
I20260812 07:00:30.841136 2998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:30.841161 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:30.827199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.27.110.254:46571)
0812 07:00:30.827718 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:30.827734 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:30.827771 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:00:30.828424 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:30.828442 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:30.828802 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:30.829161 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:30.830065 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:30.831206 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:30.832058 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:30.832546 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:30.832717 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:30.835693 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:30.835698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:30.835712 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:30.835740 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:30.837715 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:30.838296 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:30.838301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:30.838306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:30.838421 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:30.838711 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:30.838713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:30.838714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:30.839208 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:30.839408 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:30.839758 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 07:00:30.840266 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":147,"threads_started":1}
I20260812 07:00:30.841670 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f6c73b6197045bc82635f2036cdfcc7. Latest consensus state: current_term: 1 leader_uuid: "8f6c73b6197045bc82635f2036cdfcc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6c73b6197045bc82635f2036cdfcc7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46571 } } }
I20260812 07:00:30.841789 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:30.842175 3010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f6c73b6197045bc82635f2036cdfcc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f6c73b6197045bc82635f2036cdfcc7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46571 } } }
I20260812 07:00:30.842281 3010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7 [sys.catalog]: This master's current role is: LEADER
W20260812 07:00:30.846717 3031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:30.846807 3031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:30.846891 3012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:30.847802 3012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:30.853631 3012 catalog_manager.cc:1347] Generated new cluster ID: 4167c254bd644a26827e54c7904efc6a
I20260812 07:00:30.853741 3012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:30.888083 3012 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:30.889451 3012 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:30.898095 3012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f6c73b6197045bc82635f2036cdfcc7: Generated new TSK 0
I20260812 07:00:30.898955 3012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:30.946861 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:46571
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:31.065268 3041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:31.065549 3041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:31.065603 3041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:31.069778 3041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:31.069849 3041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:31.069909 3041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:31.075649 3041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:31.076707 3041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:31.077728 3041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:31.087675 3055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:31.087675 3057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:31.087808 3041 server_base.cc:1034] running on GCE node
W20260812 07:00:31.087704 3065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:31.088413 3041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:31.089053 3041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:31.090243 3041 hybrid_clock.cc:648] HybridClock initialized: now 1786518031090226 us; error 26 us; skew 500 ppm
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:31.093307 3041 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:31.094254 3041 webserver.cc:460] Webserver started at http://127.27.110.193:37905/ using document root <none> and password file <none>
I20260812 07:00:31.094826 3041 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:31.094897 3041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:31.095305 3041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:31.097406 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "f0a177924fed47d38069463a9302908d"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "e9bae065f29e34c34a91a3fd7b6f689b"
server_key_iv: "023c7192c4c5014dc0ba186deba0d712"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.097980 3041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "f0a177924fed47d38069463a9302908d"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "e9bae065f29e34c34a91a3fd7b6f689b"
server_key_iv: "023c7192c4c5014dc0ba186deba0d712"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.101755 3041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:00:31.104310 3075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.105955 3041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:00:31.106091 3041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "f0a177924fed47d38069463a9302908d"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "e9bae065f29e34c34a91a3fd7b6f689b"
server_key_iv: "023c7192c4c5014dc0ba186deba0d712"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.106218 3041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:31.117869 3041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:31.121181 3041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:31.121346 3041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:31.121897 3041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:31.122704 3041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:31.122752 3041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.122793 3041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:31.122816 3041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.132095 3041 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:38123
I20260812 07:00:31.132190 3306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:38123 every 8 connection(s)
I20260812 07:00:31.133064 3041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:31.134466 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3041
I20260812 07:00:31.134600 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:31.134931 28091 external_mini_cluster.cc:1428] Setting key c390ca4fd8b41ee960bb89d7514542b1
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:31.149235 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.135125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:33287 (local address 127.27.110.254:46571)
0812 07:00:31.135304 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:31.135310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:31.136247 (+ 937us) server_negotiation.cc:407] Connection header received
0812 07:00:31.138380 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:31.138383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:31.138429 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:31.138508 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:31.139907 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.140799 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.142132 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.142439 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.142553 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.145345 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:31.145349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:31.145350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:31.145374 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:31.146927 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.147511 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.147514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.147516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.147602 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.147983 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.147987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.147988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.148233 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:31.148352 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:31.148914 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 07:00:31.149045 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:00:31.150007 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.135572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:46571 (local address 127.27.110.193:33287)
0812 07:00:31.136084 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:31.136127 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:31.138119 (+ 1992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:31.138692 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:31.138704 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:31.139270 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:31.139753 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:31.139766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.140932 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.140935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:31.141987 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:31.142000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.142597 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.142601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:31.142782 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.143461 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:31.143478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:31.145171 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:31.147064 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:31.147089 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:31.147105 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:31.147397 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:31.147701 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:31.147707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:31.147710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:31.147877 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:31.148380 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:31.148386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:31.148624 (+ 238us) client_negotiation.cc:769] Sending connection context
0812 07:00:31.148853 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 07:00:31.149176 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":149,"threads_started":1}
I20260812 07:00:31.151504 3310 heartbeater.cc:347] Connected to a master server at 127.27.110.254:46571
I20260812 07:00:31.151768 3310 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:31.152289 3310 heartbeater.cc:511] Master 127.27.110.254:46571 requested a full tablet report, sending...
I20260812 07:00:31.153815 2923 ts_manager.cc:194] Registered new tserver with Master: f0a177924fed47d38069463a9302908d (127.27.110.193:38123)
I20260812 07:00:31.154917 2923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:33287
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:31.165709 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:46571
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:31.278772 3319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:31.279045 3319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:31.279177 3319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:31.283216 3319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:31.283273 3319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:31.283330 3319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:31.288787 3319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:31.289812 3319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:31.290793 3319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:31.300966 3343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:31.300958 3348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:31.300953 3338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:31.301380 3319 server_base.cc:1034] running on GCE node
I20260812 07:00:31.301782 3319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:31.302469 3319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:31.303646 3319 hybrid_clock.cc:648] HybridClock initialized: now 1786518031303630 us; error 29 us; skew 500 ppm
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:31.306600 3319 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:31.307631 3319 webserver.cc:460] Webserver started at http://127.27.110.194:38113/ using document root <none> and password file <none>
I20260812 07:00:31.308179 3319 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:31.308245 3319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:31.308419 3319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:31.310665 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "7df2cb934c56497791cace67d0299ebb"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "3e35d74309ddafda6ce620b933f76d06"
server_key_iv: "a4d6e0535755887d52b5351a0bb6ca76"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.311389 3319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "7df2cb934c56497791cace67d0299ebb"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "3e35d74309ddafda6ce620b933f76d06"
server_key_iv: "a4d6e0535755887d52b5351a0bb6ca76"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.315411 3319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:00:31.317984 3365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.319434 3319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:31.319572 3319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "7df2cb934c56497791cace67d0299ebb"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "3e35d74309ddafda6ce620b933f76d06"
server_key_iv: "a4d6e0535755887d52b5351a0bb6ca76"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.319689 3319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:31.332801 3319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:31.335846 3319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:31.336077 3319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:31.336884 3319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:31.337925 3319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:31.337978 3319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.338019 3319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:31.338043 3319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.348410 3319 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:42813
I20260812 07:00:31.348455 3563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:42813 every 8 connection(s)
I20260812 07:00:31.349344 3319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:31.352516 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3319
I20260812 07:00:31.352658 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:31.353091 28091 external_mini_cluster.cc:1428] Setting key 141ffd6923f785f046cc0a9319dd472c
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:31.366044 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.351164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:41593 (local address 127.27.110.254:46571)
0812 07:00:31.351329 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:31.351333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:31.352250 (+ 917us) server_negotiation.cc:407] Connection header received
0812 07:00:31.354271 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:31.354275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:31.354319 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:31.354412 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:31.356245 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.356766 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.358273 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.358693 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.358830 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.361879 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:31.361883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:31.361885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:31.361913 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:31.363554 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.364235 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.364238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.364239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.364321 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.364760 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.364763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.364764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.364967 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:31.365050 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:31.365785 (+ 735us) server_negotiation.cc:299] Negotiation successful
0812 07:00:31.365907 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:00:31.366961 3571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.351589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:46571 (local address 127.27.110.194:41593)
0812 07:00:31.352087 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:31.352134 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:31.354056 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:31.354651 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:31.354659 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:31.355299 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:31.356070 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:31.356095 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.356908 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.356915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:31.358030 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:31.358041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.358862 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.358865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:31.359020 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.359732 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:31.359750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:31.361683 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:31.363688 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:31.363695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:31.363710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:31.364087 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:31.364427 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:31.364430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:31.364432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:31.364637 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:31.365103 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:31.365110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:31.365481 (+ 371us) client_negotiation.cc:769] Sending connection context
0812 07:00:31.365715 (+ 234us) client_negotiation.cc:240] Negotiation successful
0812 07:00:31.366020 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:00:31.368423 3566 heartbeater.cc:347] Connected to a master server at 127.27.110.254:46571
I20260812 07:00:31.368711 3566 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:31.369228 3566 heartbeater.cc:511] Master 127.27.110.254:46571 requested a full tablet report, sending...
I20260812 07:00:31.370338 2923 ts_manager.cc:194] Registered new tserver with Master: 7df2cb934c56497791cace67d0299ebb (127.27.110.194:42813)
I20260812 07:00:31.370970 2923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:41593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:31.386700 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:46571
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:31.500512 3582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:31.500763 3582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:31.500831 3582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:31.504808 3582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:31.504889 3582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:31.504976 3582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:31.510151 3582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:46571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:31.511196 3582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:31.512224 3582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:31.521910 3596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:31.521947 3590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:31.521947 3582 server_base.cc:1034] running on GCE node
W20260812 07:00:31.521910 3592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:31.522620 3582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:31.523361 3582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:31.524544 3582 hybrid_clock.cc:648] HybridClock initialized: now 1786518031524508 us; error 47 us; skew 500 ppm
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:31.527462 3582 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:31.528358 3582 webserver.cc:460] Webserver started at http://127.27.110.195:35085/ using document root <none> and password file <none>
I20260812 07:00:31.528867 3582 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:31.528925 3582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:31.529084 3582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:31.531064 3582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "882b1075233d4e199953b25589a1c9f4"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "aaa8950c571bfe110521f341e9b5c332"
server_key_iv: "27d6dcd1d52aa196039333072feb510c"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.531659 3582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "882b1075233d4e199953b25589a1c9f4"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "aaa8950c571bfe110521f341e9b5c332"
server_key_iv: "27d6dcd1d52aa196039333072feb510c"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.535256 3582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260812 07:00:31.537572 3610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.538774 3582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:31.538995 3582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "882b1075233d4e199953b25589a1c9f4"
format_stamp: "Formatted at 2026-08-12 07:00:31 on dist-test-slave-w4v5"
server_key: "aaa8950c571bfe110521f341e9b5c332"
server_key_iv: "27d6dcd1d52aa196039333072feb510c"
server_key_version: "encryptionkey@0"
I20260812 07:00:31.539161 3582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:31.565918 3582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:31.569360 3582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:31.569581 3582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:31.570226 3582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:31.571130 3582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:31.571182 3582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.571255 3582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:31.571296 3582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:31.580565 3582 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:33257
I20260812 07:00:31.580600 3826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:33257 every 8 connection(s)
I20260812 07:00:31.581487 3582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:31.584218 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3582
I20260812 07:00:31.584369 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:31.584703 28091 external_mini_cluster.cc:1428] Setting key 8082bf267d31d43b2f0bd96bc39fe918
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:31.595666 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.583351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:34663 (local address 127.27.110.254:46571)
0812 07:00:31.583492 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:31.583496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:31.584463 (+ 967us) server_negotiation.cc:407] Connection header received
0812 07:00:31.586189 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:31.586193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:31.586251 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:31.586332 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:31.587784 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.588328 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.589506 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.589719 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.589798 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.592359 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:31.592362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:31.592364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:31.592389 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:31.593718 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.594217 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.594220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.594221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.594295 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.594600 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.594603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.594604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.594776 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:31.594853 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:31.595427 (+ 574us) server_negotiation.cc:299] Negotiation successful
0812 07:00:31.595542 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:00:31.596469 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.583720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:46571 (local address 127.27.110.195:34663)
0812 07:00:31.584250 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:31.584318 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:31.585985 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:31.586559 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:31.586569 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:31.587050 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:31.587646 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:31.587661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.588441 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.588444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:31.589394 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:31.589405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.589881 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.589884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:31.590059 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.590598 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:31.590614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:31.592204 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:31.593835 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:31.593840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:31.593851 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:31.594128 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:31.594377 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:31.594379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:31.594381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:31.594519 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:31.594870 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:31.594875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:31.595183 (+ 308us) client_negotiation.cc:769] Sending connection context
0812 07:00:31.595358 (+ 175us) client_negotiation.cc:240] Negotiation successful
0812 07:00:31.595622 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260812 07:00:31.597663 3833 heartbeater.cc:347] Connected to a master server at 127.27.110.254:46571
I20260812 07:00:31.597949 3833 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:31.598507 3833 heartbeater.cc:511] Master 127.27.110.254:46571 requested a full tablet report, sending...
I20260812 07:00:31.599594 2923 ts_manager.cc:194] Registered new tserver with Master: 882b1075233d4e199953b25589a1c9f4 (127.27.110.195:33257)
I20260812 07:00:31.600294 2923 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:34663
I20260812 07:00:31.610926 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:31.638430 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.629054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.27.110.193:38123)
0812 07:00:31.629332 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:31.629347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:31.629380 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:00:31.629431 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:31.629437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:31.629628 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:31.629755 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:31.630415 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.631308 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.632145 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.632511 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.632598 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.635017 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:31.635027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:31.635040 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:31.635064 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:31.636771 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.637179 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.637184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.637186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.637263 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.637500 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.637502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.637503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.637744 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:31.637907 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:31.638046 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 07:00:31.638223 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[2826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:31.649425 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:31.641350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.27.110.254:46571)
0812 07:00:31.641498 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:31.641502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:31.641670 (+ 168us) server_negotiation.cc:407] Connection header received
0812 07:00:31.641759 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:31.641761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:31.641804 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:31.641886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:31.642509 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.643142 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.643977 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:31.644257 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:31.644345 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:31.646552 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:31.646555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:31.646557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:31.646579 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:31.647879 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.648262 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.648264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.648265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.648333 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:31.648682 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:31.648684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:31.648685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:31.648887 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:31.648990 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:31.649218 (+ 228us) server_negotiation.cc:299] Negotiation successful
0812 07:00:31.649308 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 07:00:31.649855 2923 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60858
I20260812 07:00:31.650511 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3041
I20260812 07:00:31.671880 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3319
I20260812 07:00:31.681867 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3582
I20260812 07:00:31.699821 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2853
2026-08-12T07:00:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2513 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[3849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[3849](info): set up 2 sockets
Aug 12 07:00:31 dist-test-slave-w4v5 krb5kdc[3849](info): commencing operation
krb5kdc: starting...
W20260812 07:00:33.119889 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.387s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260812 07:00:33.275575 28091 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.156s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260812 07:00:33.459052 28091 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.183s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260812 07:00:33.564754 28091 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.106s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/test-user.keytab.
W20260812 07:00:33.778069 28091 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.213s user 0.000s sys 0.002s
Aug 12 07:00:33 dist-test-slave-w4v5 krb5kdc[3849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:33.832435 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:34191
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:34191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:33.947492 3876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:33.947767 3876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:33.947813 3876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:33.951941 3876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:33.951996 3876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:33.952016 3876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:33.952056 3876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:33.952090 3876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:33.957351 3876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:34191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:34191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:33.958444 3876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:33.959529 3876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:33.967818 3884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:33.967828 3888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:33.967996 3876 server_base.cc:1034] running on GCE node
W20260812 07:00:33.967842 3885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:33.968699 3876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:33.969815 3876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:33.970980 3876 hybrid_clock.cc:648] HybridClock initialized: now 1786518033970954 us; error 37 us; skew 500 ppm
Aug 12 07:00:33 dist-test-slave-w4v5 krb5kdc[3849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518033, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:33.977495 3876 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:33.978253 3876 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:33.978320 3876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:33.978564 3876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:33.980830 3876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "e29a615960a4482590047112449a49c6"
format_stamp: "Formatted at 2026-08-12 07:00:33 on dist-test-slave-w4v5"
server_key: "55cafbe93ac0a284102d9fa3e0b3aacc"
server_key_iv: "ed073877e6369448d176b61cd345aedc"
server_key_version: "encryptionkey@0"
I20260812 07:00:33.981417 3876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "e29a615960a4482590047112449a49c6"
format_stamp: "Formatted at 2026-08-12 07:00:33 on dist-test-slave-w4v5"
server_key: "55cafbe93ac0a284102d9fa3e0b3aacc"
server_key_iv: "ed073877e6369448d176b61cd345aedc"
server_key_version: "encryptionkey@0"
I20260812 07:00:33.985353 3876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:33.987886 3897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:33.989125 3876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:33.989283 3876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "e29a615960a4482590047112449a49c6"
format_stamp: "Formatted at 2026-08-12 07:00:33 on dist-test-slave-w4v5"
server_key: "55cafbe93ac0a284102d9fa3e0b3aacc"
server_key_iv: "ed073877e6369448d176b61cd345aedc"
server_key_version: "encryptionkey@0"
I20260812 07:00:33.989398 3876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:34.003237 3876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:34.006338 3876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:34.006587 3876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:34.014503 3876 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:34191
I20260812 07:00:34.014796 3980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:34191 every 8 connection(s)
I20260812 07:00:34.015388 3876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:34.018362 3981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:34.019966 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3876
I20260812 07:00:34.020140 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:34.020439 28091 external_mini_cluster.cc:1428] Setting key 7fe0d1c310ea88ae3a07b589ca9980e6
I20260812 07:00:34.025363 3981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6: Bootstrap starting.
I20260812 07:00:34.028017 3981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:34.028875 3981 log.cc:824] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:34.031440 3981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6: No bootstrap required, opened a new log
I20260812 07:00:34.035458 3981 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29a615960a4482590047112449a49c6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34191 } }
I20260812 07:00:34.035727 3981 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:34.035797 3981 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29a615960a4482590047112449a49c6, State: Initialized, Role: FOLLOWER
I20260812 07:00:34.036253 3981 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29a615960a4482590047112449a49c6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34191 } }
I20260812 07:00:34.036383 3981 raft_consensus.cc:388] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:34.036470 3981 raft_consensus.cc:482] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:34.036576 3981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:34.036887 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.022006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.27.110.254:34191)
0812 07:00:34.022445 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.022464 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.022507 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:00:34.023451 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.023478 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.023907 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.024292 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:34.025056 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.026523 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.027576 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.028308 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.028471 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.031586 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:34.031592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:34.031609 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:34.031642 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:34.033557 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.034046 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.034052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.034057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.034177 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.034464 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.034468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.034470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.034978 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:34.035236 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.035510 (+ 274us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.035890 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":107,"threads_started":1}
I20260812 07:00:34.037595 3981 raft_consensus.cc:504] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29a615960a4482590047112449a49c6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34191 } }
I20260812 07:00:34.037987 3981 leader_election.cc:302] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e29a615960a4482590047112449a49c6; no voters:
I20260812 07:00:34.038347 3981 leader_election.cc:288] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:34.038537 3992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:34.038774 3992 raft_consensus.cc:686] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [term 1 LEADER]: Becoming Leader. State: Replica: e29a615960a4482590047112449a49c6, State: Running, Role: LEADER
I20260812 07:00:34.039415 3992 consensus_queue.cc:237] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29a615960a4482590047112449a49c6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34191 } }
I20260812 07:00:34.039674 3981 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:34.041179 3994 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e29a615960a4482590047112449a49c6. Latest consensus state: current_term: 1 leader_uuid: "e29a615960a4482590047112449a49c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29a615960a4482590047112449a49c6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34191 } } }
I20260812 07:00:34.041219 3993 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e29a615960a4482590047112449a49c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29a615960a4482590047112449a49c6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34191 } } }
I20260812 07:00:34.041320 3994 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:34.041345 3993 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:34.041785 4021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:34.044893 4021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:34.050573 4021 catalog_manager.cc:1347] Generated new cluster ID: 2543a806ba2a4c4a88539f74f7f482cb
I20260812 07:00:34.050647 4021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:34.067984 4021 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:34.069211 4021 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:34.075048 4021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6: Generated new TSK 0
I20260812 07:00:34.075840 4021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:34.113488 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34191
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:34.230509 4038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:34.230784 4038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:34.230832 4038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:34.235167 4038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:34.235265 4038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:34.235357 4038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:34.241595 4038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:34.242993 4038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:34.244228 4038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:34.253858 4048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:34.253875 4046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:34.253914 4052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:34.254207 4038 server_base.cc:1034] running on GCE node
I20260812 07:00:34.254650 4038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:34.255374 4038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:34.256559 4038 hybrid_clock.cc:648] HybridClock initialized: now 1786518034256543 us; error 28 us; skew 500 ppm
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:34.259443 4038 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:34.260201 4038 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:34.260265 4038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:34.260432 4038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:34.262418 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "598e15e4337f465da7bb9b1d553aac13"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "6eb042ec39f73cb104aee37b9beca09c"
server_key_iv: "b71ea56108b4456ea38eeffa73690ce3"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.262938 4038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "598e15e4337f465da7bb9b1d553aac13"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "6eb042ec39f73cb104aee37b9beca09c"
server_key_iv: "b71ea56108b4456ea38eeffa73690ce3"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.266885 4038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:00:34.269693 4063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.271020 4038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:34.271204 4038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "598e15e4337f465da7bb9b1d553aac13"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "6eb042ec39f73cb104aee37b9beca09c"
server_key_iv: "b71ea56108b4456ea38eeffa73690ce3"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.271315 4038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:34.287528 4038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:34.290609 4038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:34.290827 4038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:34.291499 4038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:34.292356 4038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:34.292405 4038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.292490 4038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:34.292529 4038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.302336 4038 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:45035
I20260812 07:00:34.302366 4267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:45035 every 8 connection(s)
I20260812 07:00:34.302942 4038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:34.310813 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4038
I20260812 07:00:34.310974 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:34.311305 28091 external_mini_cluster.cc:1428] Setting key 449a68c613dd169b2e84c951b1c68ab6
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:34.317550 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.304710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:60531 (local address 127.27.110.254:34191)
0812 07:00:34.304844 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.304847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.305681 (+ 834us) server_negotiation.cc:407] Connection header received
0812 07:00:34.307106 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.307110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.307175 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.307292 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:34.308479 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.309031 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.310220 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.310482 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.310602 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.313361 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:34.313368 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:34.313373 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:34.313407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:34.315148 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.315805 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.315807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.315809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.315887 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.316311 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.316314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.316315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.316486 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:34.316564 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.317148 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.317306 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 07:00:34.318190 4273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.305072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:34191 (local address 127.27.110.193:60531)
0812 07:00:34.305521 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.305568 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.306853 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.307475 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.307482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.307845 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:34.308323 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.308336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.309175 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.309179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.310064 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.310074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.310660 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.310664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.310838 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.311509 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:34.311525 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:34.313186 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:34.315289 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:34.315297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:34.315314 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:34.315701 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:34.315981 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:34.315985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:34.315988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:34.316211 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:34.316592 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:34.316602 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:34.316894 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.317057 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.317319 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260812 07:00:34.319640 4268 heartbeater.cc:347] Connected to a master server at 127.27.110.254:34191
I20260812 07:00:34.319921 4268 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:34.320519 4268 heartbeater.cc:511] Master 127.27.110.254:34191 requested a full tablet report, sending...
I20260812 07:00:34.322098 3921 ts_manager.cc:194] Registered new tserver with Master: 598e15e4337f465da7bb9b1d553aac13 (127.27.110.193:45035)
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:34.323382 3921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:60531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:34.342116 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34191
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:34.454730 4280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:34.454964 4280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:34.455010 4280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:34.459033 4280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:34.459136 4280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:34.459206 4280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:34.464341 4280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:34.465266 4280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:34.466200 4280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:00:34.475557 4280 server_base.cc:1034] running on GCE node
W20260812 07:00:34.475522 4291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:34.475673 4290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:34.475522 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:34.476161 4280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:34.476783 4280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:34.477970 4280 hybrid_clock.cc:648] HybridClock initialized: now 1786518034477946 us; error 36 us; skew 500 ppm
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:34.480993 4280 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:34.481746 4280 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:34.481813 4280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:34.482043 4280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:34.484156 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "f0d2a59d11274f7d9872f0d812d45a63"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "581a2a980faa567eccefe46cde42b3d8"
server_key_iv: "4bbdab1edd8cf80cbf387f3f89229547"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.484830 4280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "f0d2a59d11274f7d9872f0d812d45a63"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "581a2a980faa567eccefe46cde42b3d8"
server_key_iv: "4bbdab1edd8cf80cbf387f3f89229547"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.488521 4280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:00:34.490855 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.492240 4280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:34.492376 4280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "f0d2a59d11274f7d9872f0d812d45a63"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "581a2a980faa567eccefe46cde42b3d8"
server_key_iv: "4bbdab1edd8cf80cbf387f3f89229547"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.492486 4280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:34.509534 4280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:34.512341 4280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:34.512544 4280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:34.513101 4280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:34.513896 4280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:34.513967 4280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.514038 4280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:34.514083 4280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.522951 4280 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:35191
I20260812 07:00:34.522974 4507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:35191 every 8 connection(s)
I20260812 07:00:34.523555 4280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:34.528621 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4280
I20260812 07:00:34.528762 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:34.529062 28091 external_mini_cluster.cc:1428] Setting key 723000b225807c54e6c5ce46f46899f2
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:34.538477 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.525289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:42787 (local address 127.27.110.254:34191)
0812 07:00:34.525437 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.525442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.526550 (+ 1108us) server_negotiation.cc:407] Connection header received
0812 07:00:34.527444 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.527447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.527495 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.527569 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:34.528869 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.529513 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.530931 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.531227 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.531314 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.534435 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:34.534438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:34.534440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:34.534488 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:34.536204 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.536822 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.536824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.536825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.536902 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.537327 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.537329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.537330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.537504 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:34.537581 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.538257 (+ 676us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.538357 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:00:34.539745 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.525721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:34191 (local address 127.27.110.194:42787)
0812 07:00:34.526404 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.526445 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.527264 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.527698 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.527704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.528040 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:34.528647 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.528671 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.529683 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.529687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.530804 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.530815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.531380 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.531386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.531639 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.532281 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:34.532306 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:34.534262 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:34.536324 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:34.536330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:34.536346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:34.536722 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:34.537000 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:34.537006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:34.537010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:34.537229 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:34.537615 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:34.537625 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:34.537976 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.538222 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.538588 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:34.541152 4512 heartbeater.cc:347] Connected to a master server at 127.27.110.254:34191
I20260812 07:00:34.541411 4512 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:34.541863 4512 heartbeater.cc:511] Master 127.27.110.254:34191 requested a full tablet report, sending...
I20260812 07:00:34.542959 3921 ts_manager.cc:194] Registered new tserver with Master: f0d2a59d11274f7d9872f0d812d45a63 (127.27.110.194:35191)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:00:34.543660 3921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:42787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:34.559900 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34191
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:34.674587 4522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:34.674880 4522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:34.674930 4522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:34.678942 4522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:34.679001 4522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:34.679059 4522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:34.684245 4522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:34.685333 4522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:34.686314 4522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:34.696084 4528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:34.696105 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:34.696440 4522 server_base.cc:1034] running on GCE node
W20260812 07:00:34.696084 4527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:34.696991 4522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:34.697638 4522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:34.698836 4522 hybrid_clock.cc:648] HybridClock initialized: now 1786518034698795 us; error 57 us; skew 500 ppm
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:34.701826 4522 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:34.702589 4522 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:34.702697 4522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:34.702941 4522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:34.705037 4522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "c275250696d54b24b13973883f520860"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "dbe791ff1a93f2a5ed84c34ced369235"
server_key_iv: "3d9bd8997eb31f2c993f1817496c9a66"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.705605 4522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "c275250696d54b24b13973883f520860"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "dbe791ff1a93f2a5ed84c34ced369235"
server_key_iv: "3d9bd8997eb31f2c993f1817496c9a66"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.709378 4522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:00:34.711992 4544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.713356 4522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:34.713507 4522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "c275250696d54b24b13973883f520860"
format_stamp: "Formatted at 2026-08-12 07:00:34 on dist-test-slave-w4v5"
server_key: "dbe791ff1a93f2a5ed84c34ced369235"
server_key_iv: "3d9bd8997eb31f2c993f1817496c9a66"
server_key_version: "encryptionkey@0"
I20260812 07:00:34.713626 4522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:34.742076 4522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:34.745555 4522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:34.745791 4522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:34.746424 4522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:34.747378 4522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:34.747452 4522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.747524 4522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:34.747575 4522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:34.756917 4522 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44931
I20260812 07:00:34.756978 4759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44931 every 8 connection(s)
I20260812 07:00:34.757589 4522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:34.767572 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4522
I20260812 07:00:34.767730 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:34.768013 28091 external_mini_cluster.cc:1428] Setting key f1cdbbd530b9d88fc7aee966c71cb81f
Aug 12 07:00:34 dist-test-slave-w4v5 krb5kdc[3849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518034, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:34.772812 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.759726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:60667 (local address 127.27.110.254:34191)
0812 07:00:34.759925 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.759931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.761184 (+ 1253us) server_negotiation.cc:407] Connection header received
0812 07:00:34.762882 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.762886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.762948 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.763039 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:34.764304 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.764889 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.766082 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.766315 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.766428 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.769203 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:34.769207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:34.769210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:34.769239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:34.770889 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.771445 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.771448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.771449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.771526 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.771836 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.771838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.771839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.772009 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:34.772087 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.772562 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.772688 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:00:34.773622 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.760336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:34191 (local address 127.27.110.195:60667)
0812 07:00:34.761008 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.761069 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.762661 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.763243 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.763250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.763608 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:34.764156 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.764172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.765041 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.765045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.765943 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.765953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.766484 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.766488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.766660 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.767302 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:34.767319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:34.769009 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:34.771001 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:34.771005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:34.771018 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:34.771350 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:34.771608 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:34.771611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:34.771612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:34.771750 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:34.772100 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:34.772105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:34.772354 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.772505 (+ 151us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.772764 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":188,"threads_started":1}
I20260812 07:00:34.774758 4761 heartbeater.cc:347] Connected to a master server at 127.27.110.254:34191
I20260812 07:00:34.775012 4761 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:34.775563 4761 heartbeater.cc:511] Master 127.27.110.254:34191 requested a full tablet report, sending...
I20260812 07:00:34.776561 3921 ts_manager.cc:194] Registered new tserver with Master: c275250696d54b24b13973883f520860 (127.27.110.195:44931)
I20260812 07:00:34.777165 3921 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:60667
I20260812 07:00:34.782119 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:00:34.791463 3987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.784158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.27.110.254:34191)
0812 07:00:34.784289 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.784293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.784472 (+ 179us) server_negotiation.cc:407] Connection header received
0812 07:00:34.784557 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.784559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.784597 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.784659 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:34.785342 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.785924 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.786780 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.787011 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.787160 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.788471 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:34.788475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:34.788478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:34.788511 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:34.789683 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.790106 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.790131 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.790137 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.790214 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:34.790513 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:34.790516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:34.790518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:34.790751 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:34.790900 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.791176 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.791297 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:00:34.794494 3921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:34.797762 3921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:34.812609 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.807060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:45035 (local address 127.0.0.1:35094)
0812 07:00:34.807780 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.807793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.807875 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.808338 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.808344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.808363 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:34.808591 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.808600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.810325 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.810329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.811262 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.811275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.812035 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.812038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.812171 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.812213 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.812331 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.812452 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":111,"thread_start_us":101,"threads_started":1}
I20260812 07:00:34.812644 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.807428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.27.110.193:45035)
0812 07:00:34.807819 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.807825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.807851 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:00:34.808000 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.808006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.808195 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.808376 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:34.808731 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.810175 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.811371 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.811896 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.812028 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.812184 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.812379 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.812482 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":98,"threads_started":1}
I20260812 07:00:34.812670 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.806193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.27.110.194:35191)
0812 07:00:34.806915 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.806931 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.807377 (+ 446us) server_negotiation.cc:407] Connection header received
0812 07:00:34.807632 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.807638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.807844 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.807986 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:34.808679 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.809419 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.811308 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.811903 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.812057 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.812192 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.812418 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.812506 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":140,"threads_started":1}
I20260812 07:00:34.813027 4781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.806037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:35191 (local address 127.0.0.1:54920)
0812 07:00:34.807264 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.807279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.807452 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.808214 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.808216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.808233 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:34.808554 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.808564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.809552 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.809557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.811171 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.811182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.812055 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.812059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.812211 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.812233 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.812801 (+ 568us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.812884 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":206,"spinlock_wait_cycles":12032,"thread_start_us":72,"threads_started":1}
I20260812 07:00:34.815585 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.806267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44931 (local address 127.0.0.1:39020)
0812 07:00:34.807210 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.807260 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.807392 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.808167 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.808172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.808191 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:34.808410 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.808418 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.809950 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.809954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.811171 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.811182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.815182 (+ 4000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.815185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.815285 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.815308 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.815390 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.815459 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":67,"thread_start_us":119,"threads_started":1}
I20260812 07:00:34.815795 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.806426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.27.110.195:44931)
0812 07:00:34.806996 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.807008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.807377 (+ 369us) server_negotiation.cc:407] Connection header received
0812 07:00:34.807615 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.807623 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.807859 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.808014 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:34.808584 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.809795 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.812256 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.813780 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.813915 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.814036 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.815448 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.815557 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":115,"threads_started":1}
I20260812 07:00:34.815778 4144 tablet_service.cc:1467] Processing CreateTablet for tablet 716b09b536a140cd83b6ac8113820b8e (DEFAULT_TABLE table=test-table [id=d85ef83f1e084429a2f66e643df38f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:34.815781 4400 tablet_service.cc:1467] Processing CreateTablet for tablet 716b09b536a140cd83b6ac8113820b8e (DEFAULT_TABLE table=test-table [id=d85ef83f1e084429a2f66e643df38f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:34.816787 4144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716b09b536a140cd83b6ac8113820b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:34.816792 4400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716b09b536a140cd83b6ac8113820b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:34.822054 4637 tablet_service.cc:1467] Processing CreateTablet for tablet 716b09b536a140cd83b6ac8113820b8e (DEFAULT_TABLE table=test-table [id=d85ef83f1e084429a2f66e643df38f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:34.823196 4637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 716b09b536a140cd83b6ac8113820b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:34.823686 4791 tablet_bootstrap.cc:492] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13: Bootstrap starting.
I20260812 07:00:34.825011 4794 tablet_bootstrap.cc:492] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Bootstrap starting.
I20260812 07:00:34.826176 4791 tablet_bootstrap.cc:654] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:34.827097 4791 log.cc:824] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:34.827666 4794 tablet_bootstrap.cc:654] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:34.828438 4794 log.cc:824] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:34.829623 4801 tablet_bootstrap.cc:492] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Bootstrap starting.
I20260812 07:00:34.830540 4794 tablet_bootstrap.cc:492] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: No bootstrap required, opened a new log
I20260812 07:00:34.830744 4794 ts_tablet_manager.cc:1397] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 07:00:34.831607 4801 tablet_bootstrap.cc:654] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:34.832406 4791 tablet_bootstrap.cc:492] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13: No bootstrap required, opened a new log
I20260812 07:00:34.832602 4791 ts_tablet_manager.cc:1397] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260812 07:00:34.832836 4801 log.cc:824] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:34.834653 4801 tablet_bootstrap.cc:492] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: No bootstrap required, opened a new log
I20260812 07:00:34.834829 4801 ts_tablet_manager.cc:1397] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 07:00:34.834741 4794 raft_consensus.cc:348] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.835024 4794 raft_consensus.cc:374] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:34.835120 4794 raft_consensus.cc:729] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d2a59d11274f7d9872f0d812d45a63, State: Initialized, Role: FOLLOWER
I20260812 07:00:34.835578 4794 consensus_queue.cc:260] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.836277 4512 heartbeater.cc:503] Master 127.27.110.254:34191 was elected leader, sending a full tablet report...
I20260812 07:00:34.836637 4794 ts_tablet_manager.cc:1428] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260812 07:00:34.837041 4791 raft_consensus.cc:348] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.837253 4791 raft_consensus.cc:374] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:34.837314 4791 raft_consensus.cc:729] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598e15e4337f465da7bb9b1d553aac13, State: Initialized, Role: FOLLOWER
I20260812 07:00:34.837730 4791 consensus_queue.cc:260] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.838366 4268 heartbeater.cc:503] Master 127.27.110.254:34191 was elected leader, sending a full tablet report...
I20260812 07:00:34.838675 4791 ts_tablet_manager.cc:1428] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 07:00:34.838778 4801 raft_consensus.cc:348] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.839039 4801 raft_consensus.cc:374] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:34.839156 4801 raft_consensus.cc:729] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c275250696d54b24b13973883f520860, State: Initialized, Role: FOLLOWER
I20260812 07:00:34.839550 4801 consensus_queue.cc:260] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.840211 4761 heartbeater.cc:503] Master 127.27.110.254:34191 was elected leader, sending a full tablet report...
I20260812 07:00:34.840468 4801 ts_tablet_manager.cc:1428] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:00:34.868943 4807 raft_consensus.cc:482] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:34.869153 4807 raft_consensus.cc:504] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.870281 4807 leader_election.cc:288] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d2a59d11274f7d9872f0d812d45a63 (127.27.110.194:35191), 598e15e4337f465da7bb9b1d553aac13 (127.27.110.193:45035)
I20260812 07:00:34.874368 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.870551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:35191 (local address 127.27.110.195:56155)
0812 07:00:34.870694 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.870712 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.870813 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.871066 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.871069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.871104 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:34.871238 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.871244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.872018 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.872023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.873204 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.873215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.873964 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.873967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.874055 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.874076 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.874137 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.874200 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 07:00:34.874445 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.870676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:56155 (local address 127.27.110.194:35191)
0812 07:00:34.870813 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.870818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.870833 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:00:34.870910 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.870913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.870962 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.871049 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:34.871359 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.871902 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.873330 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.873849 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.873977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.874031 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.874177 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.874284 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:00:34.874898 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.870687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:47455 (local address 127.27.110.193:45035)
0812 07:00:34.870835 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.870840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.871131 (+ 291us) server_negotiation.cc:407] Connection header received
0812 07:00:34.871307 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.871310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.871351 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.871434 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:34.871726 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.872263 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.873569 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.874188 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.874301 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.874346 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.874664 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.874763 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:00:34.875298 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.870551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:45035 (local address 127.27.110.195:47455)
0812 07:00:34.871019 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.871030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:34.871192 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:34.871444 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:34.871447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:34.871462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:34.871582 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.871587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.872387 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.872390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.873463 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:34.873470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.874347 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.874351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:34.874479 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.874508 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 07:00:34.875045 (+ 537us) client_negotiation.cc:240] Negotiation successful
0812 07:00:34.875152 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":113,"thread_start_us":128,"threads_started":1}
I20260812 07:00:34.875226 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b09b536a140cd83b6ac8113820b8e" candidate_uuid: "c275250696d54b24b13973883f520860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d2a59d11274f7d9872f0d812d45a63" is_pre_election: true
I20260812 07:00:34.875559 4434 raft_consensus.cc:2454] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c275250696d54b24b13973883f520860 in term 0.
I20260812 07:00:34.875873 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b09b536a140cd83b6ac8113820b8e" candidate_uuid: "c275250696d54b24b13973883f520860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598e15e4337f465da7bb9b1d553aac13" is_pre_election: true
I20260812 07:00:34.876083 4186 raft_consensus.cc:2454] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c275250696d54b24b13973883f520860 in term 0.
I20260812 07:00:34.876087 4554 leader_election.cc:302] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c275250696d54b24b13973883f520860, f0d2a59d11274f7d9872f0d812d45a63; no voters:
I20260812 07:00:34.876343 4807 raft_consensus.cc:2781] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:34.876425 4807 raft_consensus.cc:482] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:34.876492 4807 raft_consensus.cc:3037] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:34.877629 4807 raft_consensus.cc:504] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.878024 4807 leader_election.cc:288] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [CANDIDATE]: Term 1 election: Requested vote from peers f0d2a59d11274f7d9872f0d812d45a63 (127.27.110.194:35191), 598e15e4337f465da7bb9b1d553aac13 (127.27.110.193:45035)
I20260812 07:00:34.878397 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b09b536a140cd83b6ac8113820b8e" candidate_uuid: "c275250696d54b24b13973883f520860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d2a59d11274f7d9872f0d812d45a63"
I20260812 07:00:34.878440 4186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "716b09b536a140cd83b6ac8113820b8e" candidate_uuid: "c275250696d54b24b13973883f520860" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "598e15e4337f465da7bb9b1d553aac13"
I20260812 07:00:34.878510 4434 raft_consensus.cc:3037] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:34.878577 4186 raft_consensus.cc:3037] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:34.879720 4186 raft_consensus.cc:2454] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c275250696d54b24b13973883f520860 in term 1.
I20260812 07:00:34.879719 4434 raft_consensus.cc:2454] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c275250696d54b24b13973883f520860 in term 1.
I20260812 07:00:34.880046 4547 leader_election.cc:302] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 598e15e4337f465da7bb9b1d553aac13, c275250696d54b24b13973883f520860; no voters:
I20260812 07:00:34.880237 4807 raft_consensus.cc:2781] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:34.880471 4807 raft_consensus.cc:686] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 1 LEADER]: Becoming Leader. State: Replica: c275250696d54b24b13973883f520860, State: Running, Role: LEADER
I20260812 07:00:34.880864 4807 consensus_queue.cc:237] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } }
I20260812 07:00:34.884253 3918 catalog_manager.cc:5495] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 reported cstate change: term changed from 0 to 1, leader changed from <none> to c275250696d54b24b13973883f520860 (127.27.110.195). New cstate: current_term: 1 leader_uuid: "c275250696d54b24b13973883f520860" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c275250696d54b24b13973883f520860" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:34.910295 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.906652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.27.110.195:44931)
0812 07:00:34.906789 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:34.906792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:34.906807 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:00:34.906939 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:34.906941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:34.906994 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:34.907121 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:34.907415 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.907911 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.908819 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:34.909029 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:34.909109 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:34.909377 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:34.909800 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:34.909891 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:34.910030 (+ 139us) server_negotiation.cc:299] Negotiation successful
0812 07:00:34.910080 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:00:34.916313 4186 raft_consensus.cc:1261] T 716b09b536a140cd83b6ac8113820b8e P 598e15e4337f465da7bb9b1d553aac13 [term 1 FOLLOWER]: Refusing update from remote peer c275250696d54b24b13973883f520860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:34.916311 4434 raft_consensus.cc:1261] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 1 FOLLOWER]: Refusing update from remote peer c275250696d54b24b13973883f520860: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:34.917045 4811 consensus_queue.cc:1035] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "598e15e4337f465da7bb9b1d553aac13" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:34.917222 4807 consensus_queue.cc:1035] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d2a59d11274f7d9872f0d812d45a63" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 35191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:34.923383 4815 mvcc.cc:205] Tried to move back new op lower bound from 7317577871009558528 to 7317577870874677248. Current Snapshot: MvccSnapshot[applied={T|T < 7317577871009558528}]
I20260812 07:00:34.934365 3919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:34.934541 3919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:34.937065 3919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e29a615960a4482590047112449a49c6: Sending DeleteTablet for 3 replicas of tablet 716b09b536a140cd83b6ac8113820b8e
I20260812 07:00:34.937927 4400 tablet_service.cc:1514] Processing DeleteTablet for tablet 716b09b536a140cd83b6ac8113820b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:34 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:54920
I20260812 07:00:34.937994 4144 tablet_service.cc:1514] Processing DeleteTablet for tablet 716b09b536a140cd83b6ac8113820b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:34 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:35094
I20260812 07:00:34.938493 4835 tablet_replica.cc:330] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: stopping tablet replica
I20260812 07:00:34.938504 4637 tablet_service.cc:1514] Processing DeleteTablet for tablet 716b09b536a140cd83b6ac8113820b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:34 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:39020
I20260812 07:00:34.938951 4835 raft_consensus.cc:2229] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:34.938988 4839 tablet_replica.cc:330] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: stopping tablet replica
I20260812 07:00:34.939092 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4038
I20260812 07:00:34.939395 4835 raft_consensus.cc:2258] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:34.942497 4839 raft_consensus.cc:2229] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:34.942873 4839 raft_consensus.cc:2258] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:34.944248 4835 ts_tablet_manager.cc:1905] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:34.944579 4839 ts_tablet_manager.cc:1905] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:34.947645 4839 ts_tablet_manager.cc:1918] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:34.947754 4839 log.cc:1196] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/716b09b536a140cd83b6ac8113820b8e
I20260812 07:00:34.948062 4839 ts_tablet_manager.cc:1939] T 716b09b536a140cd83b6ac8113820b8e P c275250696d54b24b13973883f520860: Deleting consensus metadata
I20260812 07:00:34.949534 3899 catalog_manager.cc:4841] TS c275250696d54b24b13973883f520860 (127.27.110.195:44931): tablet 716b09b536a140cd83b6ac8113820b8e (table test-table [id=d85ef83f1e084429a2f66e643df38f32]) successfully deleted
I20260812 07:00:34.950970 4835 ts_tablet_manager.cc:1918] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:34.951109 4835 log.cc:1196] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/716b09b536a140cd83b6ac8113820b8e
I20260812 07:00:34.951447 4835 ts_tablet_manager.cc:1939] T 716b09b536a140cd83b6ac8113820b8e P f0d2a59d11274f7d9872f0d812d45a63: Deleting consensus metadata
I20260812 07:00:34.952668 3901 catalog_manager.cc:4841] TS f0d2a59d11274f7d9872f0d812d45a63 (127.27.110.194:35191): tablet 716b09b536a140cd83b6ac8113820b8e (table test-table [id=d85ef83f1e084429a2f66e643df38f32]) successfully deleted
W20260812 07:00:34.954855 3899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:45035 (error 108)
I20260812 07:00:34.955767 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4280
I20260812 07:00:34.955986 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:34.955668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:45035 (local address 127.0.0.1:35106)
0812 07:00:34.955799 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:34.955882 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 07:00:34.963246 3899 catalog_manager.cc:4584] TS 598e15e4337f465da7bb9b1d553aac13 (127.27.110.193:45035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 716b09b536a140cd83b6ac8113820b8e: Network error: Client connection negotiation failed: client connection to 127.27.110.193:45035: connect: Connection refused (error 111)
I20260812 07:00:34.989670 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4522
I20260812 07:00:35.001758 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3876
2026-08-12T07:00:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3300 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T07:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:35Z Disabled control of system clock
I20260812 07:00:35.046211 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39935
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 07:00:35.159586 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:35.159857 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:35.159894 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:35.163750 4850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:35.163800 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:35.163816 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:35.163832 4850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:35.163848 4850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:35.168401 4850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:35.169368 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:35.170753 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:35.179005 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:35.178969 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.179037 4850 server_base.cc:1034] running on GCE node
W20260812 07:00:35.178987 4865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.179873 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:35.180960 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:35.182149 4850 hybrid_clock.cc:648] HybridClock initialized: now 1786518035182113 us; error 50 us; skew 500 ppm
I20260812 07:00:35.183990 4850 webserver.cc:460] Webserver started at http://127.27.110.254:40091/ using document root <none> and password file <none>
I20260812 07:00:35.184545 4850 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:35.184643 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:35.184864 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:35.186975 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "43756df4494b433eb47f9abe68065467"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "816917db7dc0e628e8c7c3a2e43af139"
server_key_iv: "6539b7ccc4d23cf103e847e05e9eaba2"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.187578 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "43756df4494b433eb47f9abe68065467"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "816917db7dc0e628e8c7c3a2e43af139"
server_key_iv: "6539b7ccc4d23cf103e847e05e9eaba2"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.191154 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:00:35.193486 4875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.194653 4850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:00:35.194797 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "43756df4494b433eb47f9abe68065467"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "816917db7dc0e628e8c7c3a2e43af139"
server_key_iv: "6539b7ccc4d23cf103e847e05e9eaba2"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.194906 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:35.199326 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:35.199908 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:35.200083 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:35.206780 4850 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39935
I20260812 07:00:35.206786 4965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39935 every 8 connection(s)
I20260812 07:00:35.207710 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:35.210956 4966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:35.211854 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4850
I20260812 07:00:35.211987 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:35.212261 28091 external_mini_cluster.cc:1428] Setting key ab433df157eacc02c2ede988ce10db13
I20260812 07:00:35.217780 4966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467: Bootstrap starting.
I20260812 07:00:35.218302 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.213639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39440 (local address 127.27.110.254:39935)
0812 07:00:35.214333 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.214354 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.214398 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:00:35.215270 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.215293 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.215740 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.216113 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.216825 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.216833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.216848 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.216860 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.216925 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.216947 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.217087 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.217157 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.217408 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260812 07:00:35.220342 4966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:35.221127 4966 log.cc:824] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:35.223014 4966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467: No bootstrap required, opened a new log
I20260812 07:00:35.226660 4966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43756df4494b433eb47f9abe68065467" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39935 } }
I20260812 07:00:35.226892 4966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:35.226936 4966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43756df4494b433eb47f9abe68065467, State: Initialized, Role: FOLLOWER
I20260812 07:00:35.227368 4966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43756df4494b433eb47f9abe68065467" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39935 } }
I20260812 07:00:35.227532 4966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:35.227586 4966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:35.227658 4966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:35.228986 4966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43756df4494b433eb47f9abe68065467" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39935 } }
I20260812 07:00:35.229398 4966 leader_election.cc:302] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43756df4494b433eb47f9abe68065467; no voters:
I20260812 07:00:35.229733 4966 leader_election.cc:288] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:35.229884 4975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:35.230167 4975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [term 1 LEADER]: Becoming Leader. State: Replica: 43756df4494b433eb47f9abe68065467, State: Running, Role: LEADER
I20260812 07:00:35.230499 4975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43756df4494b433eb47f9abe68065467" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39935 } }
I20260812 07:00:35.231062 4966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:35.232268 4980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43756df4494b433eb47f9abe68065467. Latest consensus state: current_term: 1 leader_uuid: "43756df4494b433eb47f9abe68065467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43756df4494b433eb47f9abe68065467" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39935 } } }
I20260812 07:00:35.232396 4980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:35.232760 4976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43756df4494b433eb47f9abe68065467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43756df4494b433eb47f9abe68065467" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39935 } } }
I20260812 07:00:35.232935 4976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:35.233273 4993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:35.236137 4993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:35.241627 4993 catalog_manager.cc:1347] Generated new cluster ID: a2e210778a36436ead202846e18d1ba5
I20260812 07:00:35.241709 4993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:35.251377 4993 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:35.251681 4993 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:35.260456 4993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467: Generated new TSK 0
I20260812 07:00:35.261337 4993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:00:35.271219 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39935
--builtin_ntp_servers=127.27.110.212:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 07:00:35.386549 5009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:35.386861 5009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:35.386904 5009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:35.390933 5009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:35.390983 5009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:35.391037 5009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:35.395767 5009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:35.396782 5009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:35.398278 5009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:35.407881 5017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:35.407881 5020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.407990 5009 server_base.cc:1034] running on GCE node
W20260812 07:00:35.407882 5018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.408597 5009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:35.409215 5009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:35.410404 5009 hybrid_clock.cc:648] HybridClock initialized: now 1786518035410391 us; error 24 us; skew 500 ppm
I20260812 07:00:35.412329 5009 webserver.cc:460] Webserver started at http://127.27.110.193:39017/ using document root <none> and password file <none>
I20260812 07:00:35.412907 5009 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:35.413002 5009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:35.413237 5009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:35.415333 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "570caf323fef4eba9129651a53ef5ed2"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "99f16f7b9e4bb75840600c571ed8df19"
server_key_iv: "1fa8efda311cedee9f0ff4e591be9597"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.415903 5009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "570caf323fef4eba9129651a53ef5ed2"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "99f16f7b9e4bb75840600c571ed8df19"
server_key_iv: "1fa8efda311cedee9f0ff4e591be9597"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.419502 5009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:00:35.422020 5028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.423267 5009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:35.423396 5009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "570caf323fef4eba9129651a53ef5ed2"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "99f16f7b9e4bb75840600c571ed8df19"
server_key_iv: "1fa8efda311cedee9f0ff4e591be9597"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.423512 5009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:35.428020 5009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:35.428668 5009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:35.428848 5009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:35.429381 5009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:35.430326 5009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:35.430395 5009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.430477 5009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:35.430518 5009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.439723 5009 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:41671
I20260812 07:00:35.439749 5237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:41671 every 8 connection(s)
I20260812 07:00:35.440586 5009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:35.446661 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.442378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:39711 (local address 127.27.110.254:39935)
0812 07:00:35.442520 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.442523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.443608 (+ 1085us) server_negotiation.cc:407] Connection header received
0812 07:00:35.445018 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.445020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.445059 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.445145 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.446266 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.446270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.446272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.446285 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.446307 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.446315 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.446393 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.446514 (+ 121us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.446553 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:00:35.447561 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.442730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39935 (local address 127.27.110.193:39711)
0812 07:00:35.443429 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:35.443469 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:35.444828 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:35.445298 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:35.445305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:35.445391 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:35.445998 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:35.445999 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:35.446042 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:35.446111 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:35.446391 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:35.446398 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:35.446402 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 07:00:35.446564 (+ 162us) client_negotiation.cc:240] Negotiation successful
0812 07:00:35.446749 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":103,"threads_started":1}
I20260812 07:00:35.448745 5239 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39935
I20260812 07:00:35.448809 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 5009
I20260812 07:00:35.448961 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:35.448979 5239 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:35.449227 28091 external_mini_cluster.cc:1428] Setting key b3db4551b4619d726a4a267d34f2f533
I20260812 07:00:35.449397 5239 heartbeater.cc:511] Master 127.27.110.254:39935 requested a full tablet report, sending...
I20260812 07:00:35.451045 4904 ts_manager.cc:194] Registered new tserver with Master: 570caf323fef4eba9129651a53ef5ed2 (127.27.110.193:41671)
I20260812 07:00:35.451285 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39935
--builtin_ntp_servers=127.27.110.212:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 07:00:35.564319 5244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:35.564570 5244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:35.564610 5244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:35.568601 5244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:35.568648 5244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:35.568743 5244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:35.573396 5244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:35.574415 5244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:35.575811 5244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:35.585409 5250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.585434 5244 server_base.cc:1034] running on GCE node
W20260812 07:00:35.585449 5251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:35.585453 5253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.586035 5244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:35.586655 5244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:35.587867 5244 hybrid_clock.cc:648] HybridClock initialized: now 1786518035587820 us; error 57 us; skew 500 ppm
I20260812 07:00:35.590140 5244 webserver.cc:460] Webserver started at http://127.27.110.194:37365/ using document root <none> and password file <none>
I20260812 07:00:35.590731 5244 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:35.590795 5244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:35.591042 5244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:35.593055 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "c877a341c8514682af4110dfa9d8ee60"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "e411d3ae7a420a10b2fd96fdada3d8c8"
server_key_iv: "3d4e108277af8b5a549ec84246779630"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.593590 5244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "c877a341c8514682af4110dfa9d8ee60"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "e411d3ae7a420a10b2fd96fdada3d8c8"
server_key_iv: "3d4e108277af8b5a549ec84246779630"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.597162 5244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 07:00:35.599879 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.601073 5244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:35.601197 5244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "c877a341c8514682af4110dfa9d8ee60"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "e411d3ae7a420a10b2fd96fdada3d8c8"
server_key_iv: "3d4e108277af8b5a549ec84246779630"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.601302 5244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:35.606112 5244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:35.606706 5244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:35.606891 5244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:35.607502 5244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:35.608388 5244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:35.608436 5244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.608511 5244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:35.608551 5244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.617429 5244 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:34269
I20260812 07:00:35.617460 5448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:34269 every 8 connection(s)
I20260812 07:00:35.618314 5244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:35.624495 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.620136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:33731 (local address 127.27.110.254:39935)
0812 07:00:35.620299 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.620303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.621373 (+ 1070us) server_negotiation.cc:407] Connection header received
0812 07:00:35.622672 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.622675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.622715 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.622793 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.624065 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.624068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.624070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.624082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.624102 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.624111 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.624189 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.624313 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.624351 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 07:00:35.625427 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.620563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39935 (local address 127.27.110.194:33731)
0812 07:00:35.621240 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:35.621281 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:35.622455 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:35.622954 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:35.622961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:35.623054 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:35.623804 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:35.623806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:35.623845 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:35.623913 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:35.624189 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:35.624196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:35.624200 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 07:00:35.624327 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 07:00:35.624507 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260812 07:00:35.626618 5456 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39935
I20260812 07:00:35.626847 5456 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:35.627321 5456 heartbeater.cc:511] Master 127.27.110.254:39935 requested a full tablet report, sending...
I20260812 07:00:35.628265 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 5244
I20260812 07:00:35.628381 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:35.628532 4904 ts_manager.cc:194] Registered new tserver with Master: c877a341c8514682af4110dfa9d8ee60 (127.27.110.194:34269)
I20260812 07:00:35.628705 28091 external_mini_cluster.cc:1428] Setting key ce3bf9845068203a98d7bcd78789f2e2
I20260812 07:00:35.630748 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39935
--builtin_ntp_servers=127.27.110.212:38653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 07:00:35.743701 5471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:35.743974 5471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:35.744045 5471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:35.748003 5471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:35.748073 5471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:35.748157 5471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:35.752833 5471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:35.753855 5471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:35.755272 5471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:35.764955 5480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:35.764950 5482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:35.764971 5479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:35.765630 5471 server_base.cc:1034] running on GCE node
I20260812 07:00:35.766042 5471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:35.766664 5471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:35.767863 5471 hybrid_clock.cc:648] HybridClock initialized: now 1786518035767823 us; error 53 us; skew 500 ppm
I20260812 07:00:35.769793 5471 webserver.cc:460] Webserver started at http://127.27.110.195:33447/ using document root <none> and password file <none>
I20260812 07:00:35.770395 5471 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:35.770514 5471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:35.770752 5471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:35.772931 5471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "082c419da18944b3a0b4ab0d82acac94"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "f7a6bd2aa0ab395621329837606859c1"
server_key_iv: "4c959a007ba91ac10306cb9bd05d2fbe"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.773527 5471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "082c419da18944b3a0b4ab0d82acac94"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "f7a6bd2aa0ab395621329837606859c1"
server_key_iv: "4c959a007ba91ac10306cb9bd05d2fbe"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.777520 5471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:35.780241 5489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.781867 5471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:35.782012 5471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "082c419da18944b3a0b4ab0d82acac94"
format_stamp: "Formatted at 2026-08-12 07:00:35 on dist-test-slave-w4v5"
server_key: "f7a6bd2aa0ab395621329837606859c1"
server_key_iv: "4c959a007ba91ac10306cb9bd05d2fbe"
server_key_version: "encryptionkey@0"
I20260812 07:00:35.782135 5471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:35.786898 5471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:35.787673 5471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:35.787864 5471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:35.788422 5471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:35.789352 5471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:35.789423 5471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.789489 5471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:35.789533 5471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:35.799016 5471 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:42899
I20260812 07:00:35.799052 5687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:42899 every 8 connection(s)
I20260812 07:00:35.800015 5471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:35.807142 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.802348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:48465 (local address 127.27.110.254:39935)
0812 07:00:35.802541 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.802545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.803785 (+ 1240us) server_negotiation.cc:407] Connection header received
0812 07:00:35.805264 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.805267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.805314 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.805401 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.806586 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.806589 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.806592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.806606 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.806635 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.806644 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.806754 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.806921 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.806970 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 07:00:35.807996 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 5471
I20260812 07:00:35.808125 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:35.808200 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.802797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39935 (local address 127.27.110.195:48465)
0812 07:00:35.803590 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:35.803633 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:35.805042 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:35.805629 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:35.805637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:35.805738 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:35.806303 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:35.806304 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:35.806347 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:35.806430 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:35.806757 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:35.806770 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:35.806780 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 07:00:35.806965 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 07:00:35.807263 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":135,"threads_started":1}
I20260812 07:00:35.808444 28091 external_mini_cluster.cc:1428] Setting key dd8c97008a81137c0b18b21d4a4273eb
I20260812 07:00:35.809502 5690 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39935
I20260812 07:00:35.809751 5690 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:35.810207 5690 heartbeater.cc:511] Master 127.27.110.254:39935 requested a full tablet report, sending...
I20260812 07:00:35.811275 4906 ts_manager.cc:194] Registered new tserver with Master: 082c419da18944b3a0b4ab0d82acac94 (127.27.110.195:42899)
I20260812 07:00:35.822746 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:00:35.826836 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.824913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.27.110.254:39935)
0812 07:00:35.825064 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.825071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.825276 (+ 205us) server_negotiation.cc:407] Connection header received
0812 07:00:35.825442 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.825445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.825493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.825579 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.826376 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.826380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.826382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.826395 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.826423 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.826431 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.826525 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.826597 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.826644 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:00:35.829928 4906 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:35.833243 4906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:35.846385 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.842613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:41671 (local address 127.0.0.1:50294)
0812 07:00:35.843756 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:35.843810 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:35.843929 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:35.844501 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:35.844508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:35.844597 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:35.845356 (+ 759us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:35.845358 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:35.845381 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:35.845464 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:35.846185 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:35.846187 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:35.846190 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 07:00:35.846239 (+ 49us) client_negotiation.cc:240] Negotiation successful
0812 07:00:35.846264 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":110,"threads_started":1}
I20260812 07:00:35.846630 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.842719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.27.110.195:42899)
0812 07:00:35.843330 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.843343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.844119 (+ 776us) server_negotiation.cc:407] Connection header received
0812 07:00:35.844292 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.844297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.844443 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.844549 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.845889 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.845896 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.845907 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.845925 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.845968 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.846001 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.846143 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.846208 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.846284 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":198,"threads_started":1}
I20260812 07:00:35.846728 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.843337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.27.110.194:34269)
0812 07:00:35.844639 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.844653 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.844680 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:00:35.844737 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.844744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.844947 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.845100 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.845775 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.845780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.845789 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.845798 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.845859 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.845888 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.846006 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.846471 (+ 465us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.846562 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1149,"thread_start_us":104,"threads_started":1}
I20260812 07:00:35.846386 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.842613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:42899 (local address 127.0.0.1:35622)
0812 07:00:35.843996 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:35.844012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:35.844179 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:35.845301 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:35.845306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:35.845322 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:35.845710 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:35.845712 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:35.845724 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:35.845752 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:35.846111 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:35.846115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:35.846118 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 07:00:35.846187 (+ 69us) client_negotiation.cc:240] Negotiation successful
0812 07:00:35.846225 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":26,"thread_start_us":97,"threads_started":1}
I20260812 07:00:35.847069 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.843522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.27.110.193:41671)
0812 07:00:35.844002 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:35.844016 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:35.844047 (+ 31us) server_negotiation.cc:407] Connection header received
0812 07:00:35.844111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:35.844117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:35.844334 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:35.844481 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:35.845970 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:35.845984 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:35.845996 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:35.846008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:35.846049 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:35.846079 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:35.846797 (+ 718us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:35.846849 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 07:00:35.846916 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":129,"threads_started":1}
I20260812 07:00:35.846397 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:35.842949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:34269 (local address 127.0.0.1:55566)
0812 07:00:35.844214 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:35.844234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:35.844320 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:35.845085 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:35.845088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:35.845102 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:35.845604 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:35.845606 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:35.845616 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:35.845637 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:35.845986 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:35.845996 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:35.846003 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:00:35.846126 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 07:00:35.846204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"thread_start_us":661,"threads_started":1}
I20260812 07:00:35.850217 5111 tablet_service.cc:1467] Processing CreateTablet for tablet d3296e8d6d114a89a045e194fcefd053 (DEFAULT_TABLE table=test-table [id=e95a7ba06bd54504864cf8d80c119897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:35.850183 5571 tablet_service.cc:1467] Processing CreateTablet for tablet d3296e8d6d114a89a045e194fcefd053 (DEFAULT_TABLE table=test-table [id=e95a7ba06bd54504864cf8d80c119897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:35.851467 5111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3296e8d6d114a89a045e194fcefd053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:35.851467 5571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3296e8d6d114a89a045e194fcefd053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:35.853096 5338 tablet_service.cc:1467] Processing CreateTablet for tablet d3296e8d6d114a89a045e194fcefd053 (DEFAULT_TABLE table=test-table [id=e95a7ba06bd54504864cf8d80c119897]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:35.854115 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3296e8d6d114a89a045e194fcefd053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:35.860219 5730 tablet_bootstrap.cc:492] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Bootstrap starting.
I20260812 07:00:35.860651 5731 tablet_bootstrap.cc:492] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Bootstrap starting.
I20260812 07:00:35.861204 5729 tablet_bootstrap.cc:492] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: Bootstrap starting.
I20260812 07:00:35.862604 5731 tablet_bootstrap.cc:654] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:35.862771 5730 tablet_bootstrap.cc:654] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:35.863399 5729 tablet_bootstrap.cc:654] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:35.863530 5731 log.cc:824] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:35.863741 5730 log.cc:824] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:35.864264 5729 log.cc:824] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:35.865751 5731 tablet_bootstrap.cc:492] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: No bootstrap required, opened a new log
I20260812 07:00:35.865973 5731 ts_tablet_manager.cc:1397] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 07:00:35.866338 5730 tablet_bootstrap.cc:492] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: No bootstrap required, opened a new log
I20260812 07:00:35.866528 5729 tablet_bootstrap.cc:492] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: No bootstrap required, opened a new log
I20260812 07:00:35.866541 5730 ts_tablet_manager.cc:1397] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260812 07:00:35.866729 5729 ts_tablet_manager.cc:1397] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260812 07:00:35.869809 5731 raft_consensus.cc:348] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:35.870069 5731 raft_consensus.cc:374] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:35.870126 5731 raft_consensus.cc:729] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c877a341c8514682af4110dfa9d8ee60, State: Initialized, Role: FOLLOWER
I20260812 07:00:35.870596 5731 consensus_queue.cc:260] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:35.871316 5456 heartbeater.cc:503] Master 127.27.110.254:39935 was elected leader, sending a full tablet report...
I20260812 07:00:35.871714 5731 ts_tablet_manager.cc:1428] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:00:35.871716 5730 raft_consensus.cc:348] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:35.871716 5729 raft_consensus.cc:348] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:35.871985 5730 raft_consensus.cc:374] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:35.871986 5729 raft_consensus.cc:374] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:35.872051 5730 raft_consensus.cc:729] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082c419da18944b3a0b4ab0d82acac94, State: Initialized, Role: FOLLOWER
I20260812 07:00:35.872041 5729 raft_consensus.cc:729] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 570caf323fef4eba9129651a53ef5ed2, State: Initialized, Role: FOLLOWER
I20260812 07:00:35.872526 5729 consensus_queue.cc:260] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:35.872526 5730 consensus_queue.cc:260] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:35.873580 5729 ts_tablet_manager.cc:1428] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 07:00:35.873580 5730 ts_tablet_manager.cc:1428] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 07:00:35.873972 5690 heartbeater.cc:503] Master 127.27.110.254:39935 was elected leader, sending a full tablet report...
I20260812 07:00:35.874181 5239 heartbeater.cc:503] Master 127.27.110.254:39935 was elected leader, sending a full tablet report...
W20260812 07:00:35.941817 5240 tablet.cc:2369] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:36.012105 5742 raft_consensus.cc:482] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:36.012346 5742 raft_consensus.cc:504] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:36.013562 5742 leader_election.cc:288] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 082c419da18944b3a0b4ab0d82acac94 (127.27.110.195:42899), c877a341c8514682af4110dfa9d8ee60 (127.27.110.194:34269)
I20260812 07:00:36.015769 5745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.013789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:42899 (local address 127.27.110.193:53525)
0812 07:00:36.014073 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:36.014088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:36.014198 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:36.014479 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:36.014482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:36.014498 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:36.015093 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:36.015098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:36.015113 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:36.015148 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:36.015447 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:36.015451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:36.015455 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 07:00:36.015560 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 07:00:36.015603 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260812 07:00:36.015800 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.013938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:53525 (local address 127.27.110.195:42899)
0812 07:00:36.014071 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.014077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.014168 (+ 91us) server_negotiation.cc:407] Connection header received
0812 07:00:36.014315 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.014318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.014365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.014455 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:36.015287 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:36.015291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:36.015294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:36.015308 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:36.015331 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:36.015345 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:36.015449 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:36.015595 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 07:00:36.015650 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:36.015882 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.014116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:42911 (local address 127.27.110.194:34269)
0812 07:00:36.014258 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.014264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.014363 (+ 99us) server_negotiation.cc:407] Connection header received
0812 07:00:36.014520 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.014522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.014554 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.014641 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:36.015465 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:36.015468 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:36.015471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:36.015485 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:36.015510 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:36.015521 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:36.015608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:36.015718 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 07:00:36.015754 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:00:36.015971 5746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.013890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:34269 (local address 127.27.110.193:42911)
0812 07:00:36.014257 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:36.014272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:36.014401 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:36.014661 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:36.014664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:36.014680 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:36.015258 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:00:36.015260 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:36.015271 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:00:36.015304 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:36.015629 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:36.015633 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:36.015636 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 07:00:36.015725 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 07:00:36.015759 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":99,"threads_started":1}
I20260812 07:00:36.016582 5607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3296e8d6d114a89a045e194fcefd053" candidate_uuid: "570caf323fef4eba9129651a53ef5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082c419da18944b3a0b4ab0d82acac94" is_pre_election: true
I20260812 07:00:36.016587 5371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3296e8d6d114a89a045e194fcefd053" candidate_uuid: "570caf323fef4eba9129651a53ef5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c877a341c8514682af4110dfa9d8ee60" is_pre_election: true
I20260812 07:00:36.016870 5607 raft_consensus.cc:2454] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 570caf323fef4eba9129651a53ef5ed2 in term 0.
I20260812 07:00:36.016897 5371 raft_consensus.cc:2454] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 570caf323fef4eba9129651a53ef5ed2 in term 0.
I20260812 07:00:36.017345 5033 leader_election.cc:302] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082c419da18944b3a0b4ab0d82acac94, 570caf323fef4eba9129651a53ef5ed2; no voters:
I20260812 07:00:36.017635 5742 raft_consensus.cc:2781] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:36.017728 5742 raft_consensus.cc:482] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:36.017773 5742 raft_consensus.cc:3037] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:36.018901 5742 raft_consensus.cc:504] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:36.019305 5742 leader_election.cc:288] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [CANDIDATE]: Term 1 election: Requested vote from peers 082c419da18944b3a0b4ab0d82acac94 (127.27.110.195:42899), c877a341c8514682af4110dfa9d8ee60 (127.27.110.194:34269)
I20260812 07:00:36.019647 5607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3296e8d6d114a89a045e194fcefd053" candidate_uuid: "570caf323fef4eba9129651a53ef5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082c419da18944b3a0b4ab0d82acac94"
I20260812 07:00:36.019693 5371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3296e8d6d114a89a045e194fcefd053" candidate_uuid: "570caf323fef4eba9129651a53ef5ed2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c877a341c8514682af4110dfa9d8ee60"
I20260812 07:00:36.019798 5607 raft_consensus.cc:3037] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:36.019821 5371 raft_consensus.cc:3037] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:36.020866 5607 raft_consensus.cc:2454] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 570caf323fef4eba9129651a53ef5ed2 in term 1.
I20260812 07:00:36.020893 5371 raft_consensus.cc:2454] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 570caf323fef4eba9129651a53ef5ed2 in term 1.
I20260812 07:00:36.021162 5033 leader_election.cc:302] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082c419da18944b3a0b4ab0d82acac94, 570caf323fef4eba9129651a53ef5ed2; no voters:
I20260812 07:00:36.021390 5742 raft_consensus.cc:2781] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:36.021617 5742 raft_consensus.cc:686] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [term 1 LEADER]: Becoming Leader. State: Replica: 570caf323fef4eba9129651a53ef5ed2, State: Running, Role: LEADER
I20260812 07:00:36.021947 5742 consensus_queue.cc:237] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } }
I20260812 07:00:36.025391 4903 catalog_manager.cc:5495] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 570caf323fef4eba9129651a53ef5ed2 (127.27.110.193). New cstate: current_term: 1 leader_uuid: "570caf323fef4eba9129651a53ef5ed2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "570caf323fef4eba9129651a53ef5ed2" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 41671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:36.036891 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.035482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.27.110.193:41671)
0812 07:00:36.035612 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.035615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.035633 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:00:36.035790 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.035792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.035848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.035919 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:36.036549 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:36.036552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:36.036554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:36.036564 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:36.036582 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:36.036590 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:36.036680 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:36.036761 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 07:00:36.036794 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:00:36.042634 5371 raft_consensus.cc:1261] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 1 FOLLOWER]: Refusing update from remote peer 570caf323fef4eba9129651a53ef5ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:36.042645 5607 raft_consensus.cc:1261] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 1 FOLLOWER]: Refusing update from remote peer 570caf323fef4eba9129651a53ef5ed2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:36.043404 5748 consensus_queue.cc:1035] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c877a341c8514682af4110dfa9d8ee60" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:36.043622 5742 consensus_queue.cc:1035] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "082c419da18944b3a0b4ab0d82acac94" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:36.050486 5754 mvcc.cc:205] Tried to move back new op lower bound from 7317577875623792640 to 7317577875548356608. Current Snapshot: MvccSnapshot[applied={T|T < 7317577875623792640}]
I20260812 07:00:36.052186 5757 mvcc.cc:205] Tried to move back new op lower bound from 7317577875623792640 to 7317577875548356608. Current Snapshot: MvccSnapshot[applied={T|T < 7317577875623792640}]
I20260812 07:00:36.052680 5756 mvcc.cc:205] Tried to move back new op lower bound from 7317577875623792640 to 7317577875548356608. Current Snapshot: MvccSnapshot[applied={T|T < 7317577875623792640}]
I20260812 07:00:36.059262 4904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:36.059399 4904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:36.061764 4904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43756df4494b433eb47f9abe68065467: Sending DeleteTablet for 3 replicas of tablet d3296e8d6d114a89a045e194fcefd053
I20260812 07:00:36.062552 5571 tablet_service.cc:1514] Processing DeleteTablet for tablet d3296e8d6d114a89a045e194fcefd053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:36 UTC) from {username='slave'} at 127.0.0.1:35622
I20260812 07:00:36.062563 5111 tablet_service.cc:1514] Processing DeleteTablet for tablet d3296e8d6d114a89a045e194fcefd053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:36 UTC) from {username='slave'} at 127.0.0.1:50294
I20260812 07:00:36.062655 5338 tablet_service.cc:1514] Processing DeleteTablet for tablet d3296e8d6d114a89a045e194fcefd053 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:36 UTC) from {username='slave'} at 127.0.0.1:55566
I20260812 07:00:36.063062 5769 tablet_replica.cc:330] T d3296e8d6d114a89a045e194fcefd053 P 570caf323fef4eba9129651a53ef5ed2: stopping tablet replica
I20260812 07:00:36.063408 5767 tablet_replica.cc:330] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: stopping tablet replica
I20260812 07:00:36.063426 5771 tablet_replica.cc:330] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: stopping tablet replica
I20260812 07:00:36.063733 5767 raft_consensus.cc:2229] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:36.063805 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 5009
I20260812 07:00:36.064373 5771 raft_consensus.cc:2229] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:36.064720 5771 raft_consensus.cc:2258] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:36.067840 5767 raft_consensus.cc:2258] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:36.068902 5771 ts_tablet_manager.cc:1905] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:36.072044 5771 ts_tablet_manager.cc:1918] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:36.072140 5771 log.cc:1196] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/d3296e8d6d114a89a045e194fcefd053
I20260812 07:00:36.072479 5771 ts_tablet_manager.cc:1939] T d3296e8d6d114a89a045e194fcefd053 P c877a341c8514682af4110dfa9d8ee60: Deleting consensus metadata
I20260812 07:00:36.073383 5767 ts_tablet_manager.cc:1905] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:36.073628 4881 catalog_manager.cc:4841] TS c877a341c8514682af4110dfa9d8ee60 (127.27.110.194:34269): tablet d3296e8d6d114a89a045e194fcefd053 (table test-table [id=e95a7ba06bd54504864cf8d80c119897]) successfully deleted
I20260812 07:00:36.076860 5767 ts_tablet_manager.cc:1918] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:36.076970 5767 log.cc:1196] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/d3296e8d6d114a89a045e194fcefd053
I20260812 07:00:36.077302 5767 ts_tablet_manager.cc:1939] T d3296e8d6d114a89a045e194fcefd053 P 082c419da18944b3a0b4ab0d82acac94: Deleting consensus metadata
I20260812 07:00:36.078610 4882 catalog_manager.cc:4841] TS 082c419da18944b3a0b4ab0d82acac94 (127.27.110.195:42899): tablet d3296e8d6d114a89a045e194fcefd053 (table test-table [id=e95a7ba06bd54504864cf8d80c119897]) successfully deleted
W20260812 07:00:36.080078 4877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:41671 (error 108)
I20260812 07:00:36.080901 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 5244
I20260812 07:00:36.081126 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.080807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:41671 (local address 127.0.0.1:50310)
0812 07:00:36.080919 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:36.081027 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:41671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260812 07:00:36.097909 4877 catalog_manager.cc:4584] TS 570caf323fef4eba9129651a53ef5ed2 (127.27.110.193:41671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3296e8d6d114a89a045e194fcefd053: Network error: Client connection negotiation failed: client connection to 127.27.110.193:41671: connect: Connection refused (error 111)
I20260812 07:00:36.103507 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 5471
I20260812 07:00:36.114600 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4850
2026-08-12T07:00:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1118 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 07:00:36.152483 28091 mini_oidc.cc:152] Starting JWKS server
I20260812 07:00:36.154225 28091 webserver.cc:460] Webserver started at https://127.0.0.1:33099/ using document root <none> and password file <none>
I20260812 07:00:36.154440 28091 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 07:00:36.154731 28091 webserver.cc:460] Webserver started at http://127.0.0.1:38739/ using document root <none> and password file <none>
2026-08-12T07:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:36Z Disabled control of system clock
I20260812 07:00:36.165217 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42167
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42167
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33099/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/testchainca.cert with env {}
W20260812 07:00:36.282816 5787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:36.283196 5787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:36.283259 5787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:36.287252 5787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:36.287303 5787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:36.287318 5787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:36.287335 5787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:36.287351 5787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:00:36.287366 5787 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:00:36.287380 5787 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33099/jwks.json
I20260812 07:00:36.292881 5787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33099/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:36.293956 5787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:36.294935 5787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:36.303784 5805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:36.303802 5795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:36.303791 5797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:36.303848 5787 server_base.cc:1034] running on GCE node
I20260812 07:00:36.304709 5787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:36.305886 5787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:36.307063 5787 hybrid_clock.cc:648] HybridClock initialized: now 1786518036307029 us; error 44 us; skew 500 ppm
I20260812 07:00:36.309087 5787 webserver.cc:460] Webserver started at http://127.27.110.254:39357/ using document root <none> and password file <none>
I20260812 07:00:36.309689 5787 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:36.309757 5787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:36.309983 5787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:36.312145 5787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "874354bc3d574735b791d02984c7dc33"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "3e5a1d844d5afd1365cd9287125e00f5"
server_key_iv: "23b6e508bb699ebfd1524cc95aff196d"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.312760 5787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "874354bc3d574735b791d02984c7dc33"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "3e5a1d844d5afd1365cd9287125e00f5"
server_key_iv: "23b6e508bb699ebfd1524cc95aff196d"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.316803 5787 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:36.319494 5814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:36.320818 5787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:36.320979 5787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "874354bc3d574735b791d02984c7dc33"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "3e5a1d844d5afd1365cd9287125e00f5"
server_key_iv: "23b6e508bb699ebfd1524cc95aff196d"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.321106 5787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:36.336237 5787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:36.336915 5787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:36.337116 5787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:36.344480 5787 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42167
I20260812 07:00:36.344486 5901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42167 every 8 connection(s)
I20260812 07:00:36.345438 5787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:36.348640 5902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:36.351723 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 5787
I20260812 07:00:36.351868 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:36.352185 28091 external_mini_cluster.cc:1428] Setting key 147037ae6770d7394fe7b8ad38742adf
I20260812 07:00:36.354856 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33: Bootstrap starting.
I20260812 07:00:36.357167 5902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:36.358202 5902 log.cc:824] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:36.360212 5902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33: No bootstrap required, opened a new log
I20260812 07:00:36.362412 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.353667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.27.110.254:42167)
0812 07:00:36.354396 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.354412 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.354450 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:00:36.355218 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.355237 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.355683 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.356011 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:36.356460 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.357740 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.358986 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.359558 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.359761 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:36.360630 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:36.360635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:36.360648 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:36.361031 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:36.361086 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:36.361105 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:36.361191 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:36.361374 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 07:00:36.361709 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260812 07:00:36.364181 5902 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874354bc3d574735b791d02984c7dc33" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42167 } }
I20260812 07:00:36.364470 5902 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:36.364569 5902 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874354bc3d574735b791d02984c7dc33, State: Initialized, Role: FOLLOWER
I20260812 07:00:36.365062 5902 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874354bc3d574735b791d02984c7dc33" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42167 } }
I20260812 07:00:36.365229 5902 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:36.365296 5902 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:36.365391 5902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:36.365674 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:00:36.366420 5902 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874354bc3d574735b791d02984c7dc33" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42167 } }
I20260812 07:00:36.366817 5902 leader_election.cc:302] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874354bc3d574735b791d02984c7dc33; no voters:
I20260812 07:00:36.367223 5902 leader_election.cc:288] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:36.367377 5925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:36.367659 5925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [term 1 LEADER]: Becoming Leader. State: Replica: 874354bc3d574735b791d02984c7dc33, State: Running, Role: LEADER
I20260812 07:00:36.367987 5925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874354bc3d574735b791d02984c7dc33" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42167 } }
I20260812 07:00:36.368496 5902 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:36.369632 5926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874354bc3d574735b791d02984c7dc33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874354bc3d574735b791d02984c7dc33" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42167 } } }
I20260812 07:00:36.369782 5926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:36.370087 5928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874354bc3d574735b791d02984c7dc33. Latest consensus state: current_term: 1 leader_uuid: "874354bc3d574735b791d02984c7dc33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874354bc3d574735b791d02984c7dc33" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42167 } } }
I20260812 07:00:36.370199 5928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:36.370199 5938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:36.373658 5938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:36.379309 5938 catalog_manager.cc:1347] Generated new cluster ID: 92ac343c65cf4a2ca9c8d19e0db75933
I20260812 07:00:36.379379 5938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:36.387271 5938 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:36.388214 5938 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:36.395105 5938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874354bc3d574735b791d02984c7dc33: Generated new TSK 0
I20260812 07:00:36.395788 5938 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 07:00:36.429905 5964 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 07:00:36.430052 5964 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 07:00:36.430682 5915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.405287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.27.110.254:42167)
0812 07:00:36.405407 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.405413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.405663 (+ 250us) server_negotiation.cc:407] Connection header received
0812 07:00:36.405882 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.405885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.405936 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.406039 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:00:36.406341 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.406886 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.407879 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.408112 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.408223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:36.408570 (+ 347us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:00:36.430112 (+ 21542us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33099/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 07:00:36.430416 (+ 304us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33099/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 07:00:36.430765 5915 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43800: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33099/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 07:00:36.432036 5963 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:36.405194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42167 (local address 127.0.0.1:43800)
0812 07:00:36.405538 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:36.405551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:36.405702 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:36.406071 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:36.406074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:36.406089 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:00:36.406227 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:36.406233 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.407020 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.407024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:36.407745 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:36.407753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.408233 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.408236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:36.408331 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:36.408379 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:00:36.430347 (+ 21968us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33099/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 07:00:36.430692 (+ 345us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.110.254:42167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33099/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.110.254:42167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33099/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 07:00:36.432925 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 5787
2026-08-12T07:00:36Z chronyd exiting
I20260812 07:00:36.484937 28091 test_util.cc:175] -----------------------------------------------
I20260812 07:00:36.485059 28091 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786518006756923-28091-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (334 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 07:00:36.486219 28091 mini_oidc.cc:152] Starting JWKS server
I20260812 07:00:36.487164 28091 webserver.cc:460] Webserver started at https://127.0.0.1:38495/ using document root <none> and password file <none>
I20260812 07:00:36.487289 28091 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 07:00:36.487591 28091 webserver.cc:460] Webserver started at http://127.0.0.1:35937/ using document root <none> and password file <none>
2026-08-12T07:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:36Z Disabled control of system clock
I20260812 07:00:36.497872 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:46117
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:39263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:46117
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38495/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/testchainca.cert with env {}
W20260812 07:00:36.613767 5974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:36.614074 5974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:36.614161 5974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:36.618148 5974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:36.618240 5974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:36.618273 5974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:36.618343 5974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:36.618398 5974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:00:36.618434 5974 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:00:36.618465 5974 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38495/jwks.json
I20260812 07:00:36.624006 5974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:46117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38495/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:36.625157 5974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:36.626304 5974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:36.635187 5984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:36.635135 5988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:36.635156 5974 server_base.cc:1034] running on GCE node
W20260812 07:00:36.635135 5985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:36.636062 5974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:36.637156 5974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:36.638350 5974 hybrid_clock.cc:648] HybridClock initialized: now 1786518036638333 us; error 29 us; skew 500 ppm
I20260812 07:00:36.640375 5974 webserver.cc:460] Webserver started at http://127.27.110.254:45797/ using document root <none> and password file <none>
I20260812 07:00:36.640983 5974 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:36.641080 5974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:36.641319 5974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:36.643512 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "7b64ae5f1e8f4897a03b999e1936cb3b"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "efdca455848358a5457379a8a6bfdf4c"
server_key_iv: "0383227b079f1faa7a69b1bb2edae259"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.644097 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "7b64ae5f1e8f4897a03b999e1936cb3b"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "efdca455848358a5457379a8a6bfdf4c"
server_key_iv: "0383227b079f1faa7a69b1bb2edae259"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.647790 5974 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:00:36.650218 5998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:36.651557 5974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:36.651731 5974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "7b64ae5f1e8f4897a03b999e1936cb3b"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "efdca455848358a5457379a8a6bfdf4c"
server_key_iv: "0383227b079f1faa7a69b1bb2edae259"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.651850 5974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:36.673735 5974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:36.674464 5974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:36.674677 5974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:36.681967 5974 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:46117
I20260812 07:00:36.682000 6101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:46117 every 8 connection(s)
I20260812 07:00:36.682943 5974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:36.684403 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 5974
I20260812 07:00:36.684520 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:36.684849 28091 external_mini_cluster.cc:1428] Setting key c5f68e7faea9728f6f5953828c95f566
I20260812 07:00:36.686445 6103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:36.692121 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b: Bootstrap starting.
I20260812 07:00:36.694384 6103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:36.694506 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.686315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52984 (local address 127.27.110.254:46117)
0812 07:00:36.687045 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.687067 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.687156 (+ 89us) server_negotiation.cc:407] Connection header received
0812 07:00:36.687883 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.687902 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.688332 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.688717 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:36.689139 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.690110 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.691056 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.691710 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.691878 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:36.692831 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:36.692837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:36.692849 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:36.693236 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:36.693292 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:36.693311 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:36.693423 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:36.693542 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 07:00:36.693801 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":158,"threads_started":1}
I20260812 07:00:36.695272 6103 log.cc:824] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:36.697441 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:00:36.697667 6103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b: No bootstrap required, opened a new log
I20260812 07:00:36.701258 6103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46117 } }
I20260812 07:00:36.701502 6103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:36.701591 6103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b64ae5f1e8f4897a03b999e1936cb3b, State: Initialized, Role: FOLLOWER
I20260812 07:00:36.702008 6103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46117 } }
I20260812 07:00:36.702155 6103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:36.702242 6103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:36.702363 6103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:36.703471 6103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46117 } }
I20260812 07:00:36.703807 6103 leader_election.cc:302] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b64ae5f1e8f4897a03b999e1936cb3b; no voters:
I20260812 07:00:36.704108 6103 leader_election.cc:288] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:36.704252 6109 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:36.704468 6109 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [term 1 LEADER]: Becoming Leader. State: Replica: 7b64ae5f1e8f4897a03b999e1936cb3b, State: Running, Role: LEADER
I20260812 07:00:36.704864 6109 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46117 } }
I20260812 07:00:36.705328 6103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:36.706580 6112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b64ae5f1e8f4897a03b999e1936cb3b. Latest consensus state: current_term: 1 leader_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46117 } } }
I20260812 07:00:36.706607 6110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b64ae5f1e8f4897a03b999e1936cb3b" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 46117 } } }
I20260812 07:00:36.706787 6110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:36.706781 6112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:36.707249 6124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:36.710206 6124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:36.716243 6124 catalog_manager.cc:1347] Generated new cluster ID: 898d8b3da7d34f65960105e300d39df9
I20260812 07:00:36.716326 6124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:36.734025 6124 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:36.735357 6124 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:36.739838 6124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b64ae5f1e8f4897a03b999e1936cb3b: Generated new TSK 0
I20260812 07:00:36.740531 6124 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 07:00:36.768078 6151 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 07:00:36.768179 6151 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 07:00:36.768770 6106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:36.748524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52990 (local address 127.27.110.254:46117)
0812 07:00:36.748679 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:36.748684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:36.748892 (+ 208us) server_negotiation.cc:407] Connection header received
0812 07:00:36.749029 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:36.749032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:36.749080 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:36.749179 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:00:36.749489 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.750026 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.751342 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.751611 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.751702 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:36.752053 (+ 351us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:00:36.768437 (+ 16384us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 07:00:36.768657 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 07:00:36.768851 6106 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 07:00:36.768986 6150 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:36.748431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:46117 (local address 127.0.0.1:52990)
0812 07:00:36.748767 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:36.748783 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:36.748903 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:36.749207 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:36.749212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:36.749221 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:00:36.749343 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:36.749350 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.750144 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.750148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:36.751204 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:36.751215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:36.751740 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:36.751743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:36.751846 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:36.751875 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:00:36.768615 (+ 16740us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 07:00:36.768806 (+ 191us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.110.254:46117: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38495/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260812 07:00:36.769788 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 5974
2026-08-12T07:00:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (342 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 07:00:36.828778 28091 mini_oidc.cc:152] Starting JWKS server
I20260812 07:00:36.829674 28091 webserver.cc:460] Webserver started at https://127.0.0.1:45223/ using document root <none> and password file <none>
I20260812 07:00:36.829757 28091 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 07:00:36.830015 28091 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
2026-08-12T07:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:36Z Disabled control of system clock
I20260812 07:00:36.840740 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:33627
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:33627
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45223/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 07:00:36.953675 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:36.954034 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:36.954118 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:36.958278 6163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:36.958376 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:36.958408 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:36.958446 6163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:36.958483 6163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:00:36.958521 6163 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:00:36.958555 6163 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45223/jwks.json
I20260812 07:00:36.963631 6163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:33627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:33627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45223/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:36.964856 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:36.966325 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:36.975273 6173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:36.975349 6163 server_base.cc:1034] running on GCE node
W20260812 07:00:36.975296 6169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:36.975328 6171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:36.976187 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:36.977385 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:36.978564 6163 hybrid_clock.cc:648] HybridClock initialized: now 1786518036978535 us; error 39 us; skew 500 ppm
I20260812 07:00:36.980597 6163 webserver.cc:460] Webserver started at http://127.27.110.254:42367/ using document root <none> and password file <none>
I20260812 07:00:36.981223 6163 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:36.981289 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:36.981516 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:36.983753 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "88537ac876294bd0889103b1a7dcb4fa"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "ab1f6e2ff0b169c0591c8c46b774f864"
server_key_iv: "afe30f19c75d6cae29c5a4d23878aa70"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.984359 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "88537ac876294bd0889103b1a7dcb4fa"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "ab1f6e2ff0b169c0591c8c46b774f864"
server_key_iv: "afe30f19c75d6cae29c5a4d23878aa70"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.988397 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:00:36.991279 6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:36.992703 6163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:36.992862 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "88537ac876294bd0889103b1a7dcb4fa"
format_stamp: "Formatted at 2026-08-12 07:00:36 on dist-test-slave-w4v5"
server_key: "ab1f6e2ff0b169c0591c8c46b774f864"
server_key_iv: "afe30f19c75d6cae29c5a4d23878aa70"
server_key_version: "encryptionkey@0"
I20260812 07:00:36.992995 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:37.020975 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:37.021842 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:37.022078 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:37.029786 6163 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:33627
I20260812 07:00:37.029803 6263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:33627 every 8 connection(s)
I20260812 07:00:37.031045 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:37.034543 6264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:37.037709 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 6163
I20260812 07:00:37.037863 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:37.038163 28091 external_mini_cluster.cc:1428] Setting key 81354405da9b43ea7336a66c9d5ed24e
I20260812 07:00:37.041401 6264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa: Bootstrap starting.
I20260812 07:00:37.043855 6264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:37.044646 6264 log.cc:824] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:37.046756 6264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa: No bootstrap required, opened a new log
I20260812 07:00:37.048178 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:37.039817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.27.110.254:33627)
0812 07:00:37.040527 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:37.040543 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:37.040619 (+ 76us) server_negotiation.cc:407] Connection header received
0812 07:00:37.041245 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:37.041263 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:37.041749 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:37.042108 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:37.042571 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.043765 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.044731 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.045264 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.045442 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:37.046366 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:37.046372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:37.046385 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:37.046781 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:37.046836 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:37.046883 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:37.047006 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:37.047182 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 07:00:37.047449 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":146,"threads_started":1}
I20260812 07:00:37.051170 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:00:37.050980 6264 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88537ac876294bd0889103b1a7dcb4fa" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33627 } }
I20260812 07:00:37.051286 6264 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:37.051363 6264 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88537ac876294bd0889103b1a7dcb4fa, State: Initialized, Role: FOLLOWER
I20260812 07:00:37.051811 6264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88537ac876294bd0889103b1a7dcb4fa" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33627 } }
I20260812 07:00:37.051959 6264 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:37.052024 6264 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:37.052125 6264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:37.053184 6264 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88537ac876294bd0889103b1a7dcb4fa" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33627 } }
I20260812 07:00:37.053479 6264 leader_election.cc:302] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88537ac876294bd0889103b1a7dcb4fa; no voters:
I20260812 07:00:37.053745 6264 leader_election.cc:288] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:37.053897 6273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:37.054173 6273 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [term 1 LEADER]: Becoming Leader. State: Replica: 88537ac876294bd0889103b1a7dcb4fa, State: Running, Role: LEADER
I20260812 07:00:37.054479 6273 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88537ac876294bd0889103b1a7dcb4fa" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33627 } }
I20260812 07:00:37.054955 6264 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:37.056257 6278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88537ac876294bd0889103b1a7dcb4fa. Latest consensus state: current_term: 1 leader_uuid: "88537ac876294bd0889103b1a7dcb4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88537ac876294bd0889103b1a7dcb4fa" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33627 } } }
I20260812 07:00:37.056389 6278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:37.056743 6275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88537ac876294bd0889103b1a7dcb4fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88537ac876294bd0889103b1a7dcb4fa" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33627 } } }
I20260812 07:00:37.056862 6275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:37.056838 6292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:37.060338 6292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:37.065385 6292 catalog_manager.cc:1347] Generated new cluster ID: 846d56b51d6943e5bacd2757d7b7e230
I20260812 07:00:37.065455 6292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:37.087218 6292 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:37.088536 6292 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:37.101902 6292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88537ac876294bd0889103b1a7dcb4fa: Generated new TSK 0
I20260812 07:00:37.102743 6292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:00:37.121536 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:37.116332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.27.110.254:33627)
0812 07:00:37.116471 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:37.116476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:37.116677 (+ 201us) server_negotiation.cc:407] Connection header received
0812 07:00:37.116838 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:37.116840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:37.116889 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:37.116970 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:00:37.117246 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.117782 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.119066 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.119370 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.119463 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:37.119824 (+ 361us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:00:37.121124 (+ 1300us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 07:00:37.121380 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 07:00:37.121609 6319 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:37.116259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33627 (local address 127.0.0.1:45376)
0812 07:00:37.116579 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:37.116595 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:37.116716 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:37.117006 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:37.117010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:37.117019 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:00:37.117137 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:37.117143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.117895 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.117898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:37.118912 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:37.118922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.119502 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.119507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:37.119600 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:37.119628 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:00:37.121289 (+ 1661us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 07:00:37.121435 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.110.254:33627: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
W20260812 07:00:37.122014 6268 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45376: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45223/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260812 07:00:37.122206 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 6163
E20260812 07:00:37.140161 6323 webserver.cc:533] Webserver: sslize failed:
E20260812 07:00:37.140244 6323 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-12T07:00:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (345 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T07:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:37Z Disabled control of system clock
I20260812 07:00:37.183689 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42063
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42063
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/testchainca.cert with env {}
W20260812 07:00:37.297843 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:37.298133 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:37.298177 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:37.302229 6327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:37.302284 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:37.302300 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:37.302318 6327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:37.302333 6327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 07:00:37.302349 6327 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 07:00:37.302363 6327 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 07:00:37.307642 6327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:37.308636 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:37.309669 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:37.318492 6335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:37.318545 6334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:37.318567 6327 server_base.cc:1034] running on GCE node
W20260812 07:00:37.318518 6337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:37.319453 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:37.320501 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:37.321713 6327 hybrid_clock.cc:648] HybridClock initialized: now 1786518037321690 us; error 32 us; skew 500 ppm
I20260812 07:00:37.323796 6327 webserver.cc:460] Webserver started at http://127.27.110.254:44961/ using document root <none> and password file <none>
I20260812 07:00:37.324400 6327 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:37.324467 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:37.324694 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:37.327261 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "dbe474ffe97a46b1921beeea7ed04d4d"
format_stamp: "Formatted at 2026-08-12 07:00:37 on dist-test-slave-w4v5"
server_key: "d03f2cb472b07061263e70f512aab9f9"
server_key_iv: "16ba5e921b1b1f993cee461c6e55daf9"
server_key_version: "encryptionkey@0"
I20260812 07:00:37.327854 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "dbe474ffe97a46b1921beeea7ed04d4d"
format_stamp: "Formatted at 2026-08-12 07:00:37 on dist-test-slave-w4v5"
server_key: "d03f2cb472b07061263e70f512aab9f9"
server_key_iv: "16ba5e921b1b1f993cee461c6e55daf9"
server_key_version: "encryptionkey@0"
I20260812 07:00:37.331658 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:00:37.334141 6346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:37.335492 6327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:37.335635 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "dbe474ffe97a46b1921beeea7ed04d4d"
format_stamp: "Formatted at 2026-08-12 07:00:37 on dist-test-slave-w4v5"
server_key: "d03f2cb472b07061263e70f512aab9f9"
server_key_iv: "16ba5e921b1b1f993cee461c6e55daf9"
server_key_version: "encryptionkey@0"
I20260812 07:00:37.335754 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:37.353284 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:37.353957 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:37.354161 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:37.361129 6327 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42063
I20260812 07:00:37.361146 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42063 every 8 connection(s)
I20260812 07:00:37.362095 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:37.364967 6433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:37.370169 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 6327
I20260812 07:00:37.370326 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:37.370569 28091 external_mini_cluster.cc:1428] Setting key fa15069e589a5a4b0c145adf388093d3
I20260812 07:00:37.370744 6433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d: Bootstrap starting.
I20260812 07:00:37.373201 6433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:37.374007 6433 log.cc:824] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:37.376304 6433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d: No bootstrap required, opened a new log
I20260812 07:00:37.380036 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:37.371820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.27.110.254:42063)
0812 07:00:37.372438 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:37.372452 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:37.372489 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:00:37.373107 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:37.373123 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:37.373635 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:37.373922 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:37.374315 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.375548 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.376797 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.377254 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.377372 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:37.378250 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:37.378255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:37.378268 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:00:37.378643 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:37.378708 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:00:37.378728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:37.378805 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:37.378981 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 07:00:37.379308 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
I20260812 07:00:37.380342 6433 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42063 } }
I20260812 07:00:37.380594 6433 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:37.380682 6433 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe474ffe97a46b1921beeea7ed04d4d, State: Initialized, Role: FOLLOWER
I20260812 07:00:37.381191 6433 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42063 } }
I20260812 07:00:37.381353 6433 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:37.381421 6433 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:37.381513 6433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:37.382570 6433 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42063 } }
I20260812 07:00:37.382926 6433 leader_election.cc:302] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe474ffe97a46b1921beeea7ed04d4d; no voters:
I20260812 07:00:37.383121 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:00:37.383296 6433 leader_election.cc:288] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:37.383409 6439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:37.383688 6439 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [term 1 LEADER]: Becoming Leader. State: Replica: dbe474ffe97a46b1921beeea7ed04d4d, State: Running, Role: LEADER
I20260812 07:00:37.384044 6439 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42063 } }
I20260812 07:00:37.384531 6433 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:37.385979 6442 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42063 } } }
I20260812 07:00:37.385979 6440 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe474ffe97a46b1921beeea7ed04d4d. Latest consensus state: current_term: 1 leader_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe474ffe97a46b1921beeea7ed04d4d" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42063 } } }
I20260812 07:00:37.386133 6442 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:37.386234 6440 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d [sys.catalog]: This master's current role is: LEADER
W20260812 07:00:37.389596 6460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:37.389698 6460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:37.389777 6454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:37.390697 6454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:37.396063 6454 catalog_manager.cc:1347] Generated new cluster ID: b6cc56ff9fb24f618d7fd9831ee5a074
I20260812 07:00:37.396128 6454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:37.438249 6454 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:37.439472 6454 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:37.450126 6454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbe474ffe97a46b1921beeea7ed04d4d: Generated new TSK 0
I20260812 07:00:37.450968 6454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:00:37.516369 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:37.467264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59176 (local address 127.27.110.254:42063)
0812 07:00:37.467407 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:37.467412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:37.467634 (+ 222us) server_negotiation.cc:407] Connection header received
0812 07:00:37.467781 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:37.467784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:37.467841 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:37.467959 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0812 07:00:37.468174 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.468955 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.469819 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.470167 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.470268 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:37.470600 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 07:00:37.515859 (+ 45259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 07:00:37.516209 (+ 350us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59176: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 07:00:37.516461 6436 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59176: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 07:00:37.516521 6478 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:37.467220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42063 (local address 127.0.0.1:59176)
0812 07:00:37.467546 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:37.467559 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:37.467656 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:37.467963 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:37.467965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:37.467972 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0812 07:00:37.468066 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:37.468071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.469072 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.469074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:37.469704 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:37.469712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:37.470303 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:37.470306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:37.470395 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:37.470420 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 07:00:37.516122 (+ 45702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 07:00:37.516325 (+ 203us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.110.254:42063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":153,"threads_started":1}
I20260812 07:00:37.517441 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 6327
2026-08-12T07:00:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (373 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (139 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (137 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:37 dist-test-slave-w4v5 krb5kdc[6503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:37 dist-test-slave-w4v5 krb5kdc[6503](info): set up 2 sockets
Aug 12 07:00:37 dist-test-slave-w4v5 krb5kdc[6503](info): commencing operation
krb5kdc: starting...
W20260812 07:00:39.169966 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:39 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:39.299825 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43575
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:43575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:39.411125 6523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:39.411423 6523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:39.411507 6523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:39.415475 6523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:39.415562 6523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:39.415601 6523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:39.415657 6523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:39.415697 6523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:39.420789 6523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:43575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:39.421820 6523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:39.422859 6523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:39.431280 6541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:39.431347 6536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:39.431347 6537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:39.431536 6523 server_base.cc:1034] running on GCE node
I20260812 07:00:39.432103 6523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:39.433102 6523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:39.434257 6523 hybrid_clock.cc:648] HybridClock initialized: now 1786518039434228 us; error 42 us; skew 500 ppm
Aug 12 07:00:39 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:39.442203 6523 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:39.443174 6523 webserver.cc:460] Webserver started at http://127.27.110.254:32799/ using document root <none> and password file <none>
I20260812 07:00:39.443724 6523 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:39.443816 6523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:39.444048 6523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:39.446036 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "9bccdefe3d514c958d22c426a51c58e7"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "5dddd34e6c6d47085fb12be242dc4d55"
server_key_iv: "5c0f1e877bae5e5196696b24cabd2637"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.446633 6523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "9bccdefe3d514c958d22c426a51c58e7"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "5dddd34e6c6d47085fb12be242dc4d55"
server_key_iv: "5c0f1e877bae5e5196696b24cabd2637"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.450371 6523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260812 07:00:39.452734 6550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:39.453869 6523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 07:00:39.454005 6523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "9bccdefe3d514c958d22c426a51c58e7"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "5dddd34e6c6d47085fb12be242dc4d55"
server_key_iv: "5c0f1e877bae5e5196696b24cabd2637"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.454115 6523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:39.465672 6523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:39.468726 6523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:39.468920 6523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:39.476022 6523 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:43575
I20260812 07:00:39.476050 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:43575 every 8 connection(s)
I20260812 07:00:39.476923 6523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:39.479642 6637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:39.484876 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7: Bootstrap starting.
I20260812 07:00:39.486527 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 6523
I20260812 07:00:39.486662 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:39.486931 28091 external_mini_cluster.cc:1428] Setting key 77f7f96446476d22759b01c868f6677f
I20260812 07:00:39.487555 6637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:39.488488 6637 log.cc:824] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:39.490492 6637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7: No bootstrap required, opened a new log
I20260812 07:00:39.493914 6637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bccdefe3d514c958d22c426a51c58e7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43575 } }
I20260812 07:00:39.494215 6637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:39.494272 6637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bccdefe3d514c958d22c426a51c58e7, State: Initialized, Role: FOLLOWER
I20260812 07:00:39.494773 6637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bccdefe3d514c958d22c426a51c58e7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43575 } }
I20260812 07:00:39.494932 6637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:39.495020 6637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:39.495157 6637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 0 FOLLOWER]: Advancing to term 1
Aug 12 07:00:39 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:39.496390 6637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bccdefe3d514c958d22c426a51c58e7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43575 } }
I20260812 07:00:39.496798 6637 leader_election.cc:302] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bccdefe3d514c958d22c426a51c58e7; no voters:
I20260812 07:00:39.497221 6637 leader_election.cc:288] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:39.497335 6643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:39.497608 6643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [term 1 LEADER]: Becoming Leader. State: Replica: 9bccdefe3d514c958d22c426a51c58e7, State: Running, Role: LEADER
I20260812 07:00:39.498015 6643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bccdefe3d514c958d22c426a51c58e7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43575 } }
I20260812 07:00:39.498669 6637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:39.499815 6649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bccdefe3d514c958d22c426a51c58e7. Latest consensus state: current_term: 1 leader_uuid: "9bccdefe3d514c958d22c426a51c58e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bccdefe3d514c958d22c426a51c58e7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43575 } } }
I20260812 07:00:39.500223 6649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:39.500105 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bccdefe3d514c958d22c426a51c58e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bccdefe3d514c958d22c426a51c58e7" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43575 } } }
I20260812 07:00:39.500670 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:39.503095 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:39.488181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.27.110.254:43575)
0812 07:00:39.488718 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:39.488735 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:39.488772 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:00:39.489603 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:39.489622 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:39.489933 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:39.490275 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:39.491154 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:39.492139 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:39.493006 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:39.493486 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:39.493674 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:39.496788 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:39.496793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:39.496806 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:39.496833 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:39.499066 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:39.499685 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:39.499694 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:39.499703 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:39.499864 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:39.500218 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:39.500222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:39.500224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:39.500791 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:39.501012 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:39.501828 (+ 816us) server_negotiation.cc:299] Negotiation successful
0812 07:00:39.502145 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":142,"threads_started":1}
W20260812 07:00:39.504493 6675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:39.504570 6675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:39.504639 6667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:39.505471 6667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:39.510986 6667 catalog_manager.cc:1347] Generated new cluster ID: c02ce7a21eeb48d88f25fa546dbabfbc
I20260812 07:00:39.511051 6667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:39.552531 6667 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:39.553859 6667 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:39.559680 6667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bccdefe3d514c958d22c426a51c58e7: Generated new TSK 0
I20260812 07:00:39.560463 6667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:39.595486 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43575
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:39.710848 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:39.711162 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:39.711241 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:39.715301 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:39.715359 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:39.715494 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:39.720794 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:39.721938 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:39.723114 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:39.733922 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:39.733896 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:39.733930 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:39.734225 6687 server_base.cc:1034] running on GCE node
I20260812 07:00:39.734674 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:39.735387 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:39.736578 6687 hybrid_clock.cc:648] HybridClock initialized: now 1786518039736563 us; error 25 us; skew 500 ppm
Aug 12 07:00:39 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:39.739565 6687 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:39.740460 6687 webserver.cc:460] Webserver started at http://127.27.110.193:36137/ using document root <none> and password file <none>
I20260812 07:00:39.741016 6687 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:39.741086 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:39.741252 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:39.743356 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "15f04141e3d141aeaf5e9b446e38070f"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "ca39ddfebf3d320086b010e927c4240c"
server_key_iv: "985c7e64ab637b7694b88ccb4ca421ff"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.743942 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "15f04141e3d141aeaf5e9b446e38070f"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "ca39ddfebf3d320086b010e927c4240c"
server_key_iv: "985c7e64ab637b7694b88ccb4ca421ff"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.747733 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:00:39.750296 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:39.751974 6687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:39.752126 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "15f04141e3d141aeaf5e9b446e38070f"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "ca39ddfebf3d320086b010e927c4240c"
server_key_iv: "985c7e64ab637b7694b88ccb4ca421ff"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.752244 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:39.789227 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:39.792709 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:39.792927 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:39.793581 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:39.794464 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:39.794538 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:39.794610 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:39.794661 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:39.804275 6687 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:32939
I20260812 07:00:39.804308 6904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:32939 every 8 connection(s)
I20260812 07:00:39.805200 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:39.813493 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 6687
I20260812 07:00:39.813606 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:39.813920 28091 external_mini_cluster.cc:1428] Setting key e013f7d49517182aac9a3ac30dee0e26
Aug 12 07:00:39 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:39.820989 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:39.807155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:57679 (local address 127.27.110.254:43575)
0812 07:00:39.807334 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:39.807338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:39.808195 (+ 857us) server_negotiation.cc:407] Connection header received
0812 07:00:39.809708 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:39.809712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:39.809775 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:39.809857 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:39.811128 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:39.811701 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:39.812907 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:39.813165 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:39.813279 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:39.816659 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:39.816663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:39.816665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:39.816690 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:39.818536 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:39.819124 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:39.819129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:39.819133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:39.819239 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:39.819654 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:39.819660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:39.819664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:39.819942 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:39.820054 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:39.820566 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 07:00:39.820721 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 07:00:39.821754 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:39.807558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43575 (local address 127.27.110.193:57679)
0812 07:00:39.808032 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:39.808077 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:39.809432 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:39.810040 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:39.810047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:39.810413 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:39.810933 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:39.810947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:39.811840 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:39.811850 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:39.812767 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:39.812777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:39.813338 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:39.813342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:39.813534 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:39.814334 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:39.814359 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:39.816433 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:39.818671 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:39.818683 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:39.818704 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:39.818989 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:39.819353 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:39.819356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:39.819358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:39.819504 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:39.820051 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:39.820057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:39.820307 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 07:00:39.820473 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 07:00:39.820792 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260812 07:00:39.823216 6908 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43575
I20260812 07:00:39.823505 6908 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:39.824144 6908 heartbeater.cc:511] Master 127.27.110.254:43575 requested a full tablet report, sending...
I20260812 07:00:39.825901 6575 ts_manager.cc:194] Registered new tserver with Master: 15f04141e3d141aeaf5e9b446e38070f (127.27.110.193:32939)
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:39.827235 6575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:57679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:39.845829 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43575
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:39.961808 6915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:39.962105 6915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:39.962154 6915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:39.966117 6915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:39.966177 6915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:39.966234 6915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:39.971495 6915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:39.972569 6915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:39.973537 6915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:39.983156 6925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:39.983167 6926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:39.983269 6915 server_base.cc:1034] running on GCE node
W20260812 07:00:39.983165 6931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:39.983841 6915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:39.984452 6915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:39.985615 6915 hybrid_clock.cc:648] HybridClock initialized: now 1786518039985582 us; error 42 us; skew 500 ppm
Aug 12 07:00:39 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:39.988468 6915 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:39.989341 6915 webserver.cc:460] Webserver started at http://127.27.110.194:42003/ using document root <none> and password file <none>
I20260812 07:00:39.989933 6915 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:39.989997 6915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:39.990163 6915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:39.992594 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "371bf13c52c9432ca6ab27714ea54769"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "68647e1701cbc761f443bbc3fbf00ff5"
server_key_iv: "c4b756251217ff4ab1df51fa359ef3a4"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.993122 6915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "371bf13c52c9432ca6ab27714ea54769"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "68647e1701cbc761f443bbc3fbf00ff5"
server_key_iv: "c4b756251217ff4ab1df51fa359ef3a4"
server_key_version: "encryptionkey@0"
I20260812 07:00:39.996903 6915 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 07:00:39.999300 6950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:40.000737 6915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:00:40.000834 6915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "371bf13c52c9432ca6ab27714ea54769"
format_stamp: "Formatted at 2026-08-12 07:00:39 on dist-test-slave-w4v5"
server_key: "68647e1701cbc761f443bbc3fbf00ff5"
server_key_iv: "c4b756251217ff4ab1df51fa359ef3a4"
server_key_version: "encryptionkey@0"
I20260812 07:00:40.000916 6915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:40.018740 6915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:40.021970 6915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:40.022202 6915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:40.022878 6915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:40.023777 6915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:40.023826 6915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:40.023891 6915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:40.023923 6915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:40.033236 6915 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:35099
I20260812 07:00:40.033257 7143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:35099 every 8 connection(s)
I20260812 07:00:40.034134 6915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:40.043236 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 6915
I20260812 07:00:40.043359 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:40.043665 28091 external_mini_cluster.cc:1428] Setting key 424e543d2be1ed4bde6991e9d1da25df
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518039, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:40.052732 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.036025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:33777 (local address 127.27.110.254:43575)
0812 07:00:40.036163 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.036167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.036994 (+ 827us) server_negotiation.cc:407] Connection header received
0812 07:00:40.038425 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.038428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.038489 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.038592 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.039802 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.040330 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.041638 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.042058 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.042185 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.046547 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:40.046554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:40.046560 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:40.046597 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:40.048623 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.049201 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.049206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.049211 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.049289 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.049643 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.049648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.049652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.049869 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:40.050963 (+ 1094us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:40.051806 (+ 843us) server_negotiation.cc:299] Negotiation successful
0812 07:00:40.052479 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:40.053241 7149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.036399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43575 (local address 127.27.110.194:33777)
0812 07:00:40.036843 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:40.036884 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:40.038217 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:40.038768 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:40.038775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:40.039192 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:40.039659 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:40.039673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.040458 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.040466 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:40.041488 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:40.041498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.042265 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.042269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:40.042470 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.044142 (+ 1672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:40.044164 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:40.046354 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:40.048745 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:40.048753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:40.048770 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:40.049085 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:40.049388 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:40.049394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:40.049398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:40.049551 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:40.051060 (+ 1509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:40.051096 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:40.051445 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 07:00:40.051703 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 07:00:40.052164 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260812 07:00:40.054517 7144 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43575
I20260812 07:00:40.054782 7144 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:40.055375 7144 heartbeater.cc:511] Master 127.27.110.254:43575 requested a full tablet report, sending...
I20260812 07:00:40.056571 6575 ts_manager.cc:194] Registered new tserver with Master: 371bf13c52c9432ca6ab27714ea54769 (127.27.110.194:35099)
I20260812 07:00:40.057469 6575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:33777
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:40.076205 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43575
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:40.191527 7155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:40.191847 7155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:40.191897 7155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:40.195803 7155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:40.195851 7155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:40.195909 7155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:40.201221 7155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:40.202224 7155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:40.203184 7155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:40.213280 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:40.213284 7164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:40.213466 7155 server_base.cc:1034] running on GCE node
W20260812 07:00:40.213280 7163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:40.213899 7155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:40.214519 7155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:40.215687 7155 hybrid_clock.cc:648] HybridClock initialized: now 1786518040215673 us; error 24 us; skew 500 ppm
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.218441 7155 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:40.219349 7155 webserver.cc:460] Webserver started at http://127.27.110.195:38309/ using document root <none> and password file <none>
I20260812 07:00:40.219877 7155 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:40.219935 7155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:40.220105 7155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:40.222105 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "e8ff83c3de3a45a4bc732ea7b6f43bf7"
format_stamp: "Formatted at 2026-08-12 07:00:40 on dist-test-slave-w4v5"
server_key: "e9c7954fd85013aeb28e9ce1eeebcfd5"
server_key_iv: "7ef021e188f196b3a1d7119ae1eaafcc"
server_key_version: "encryptionkey@0"
I20260812 07:00:40.222638 7155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "e8ff83c3de3a45a4bc732ea7b6f43bf7"
format_stamp: "Formatted at 2026-08-12 07:00:40 on dist-test-slave-w4v5"
server_key: "e9c7954fd85013aeb28e9ce1eeebcfd5"
server_key_iv: "7ef021e188f196b3a1d7119ae1eaafcc"
server_key_version: "encryptionkey@0"
I20260812 07:00:40.226338 7155 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:00:40.228859 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:40.230206 7155 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:40.230418 7155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "e8ff83c3de3a45a4bc732ea7b6f43bf7"
format_stamp: "Formatted at 2026-08-12 07:00:40 on dist-test-slave-w4v5"
server_key: "e9c7954fd85013aeb28e9ce1eeebcfd5"
server_key_iv: "7ef021e188f196b3a1d7119ae1eaafcc"
server_key_version: "encryptionkey@0"
I20260812 07:00:40.230558 7155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:40.252694 7155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:40.255831 7155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:40.256079 7155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:40.256721 7155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:40.257655 7155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:40.257704 7155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:40.257772 7155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:40.257809 7155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:40.267019 7155 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:39627
I20260812 07:00:40.267040 7372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:39627 every 8 connection(s)
I20260812 07:00:40.268049 7155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:40.273135 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 7155
I20260812 07:00:40.273298 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:40.273584 28091 external_mini_cluster.cc:1428] Setting key c3edbf65f27a398498a4b6cbc4c1e5ff
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:40.282145 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.270060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:36585 (local address 127.27.110.254:43575)
0812 07:00:40.270215 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.270219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.271115 (+ 896us) server_negotiation.cc:407] Connection header received
0812 07:00:40.272538 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.272541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.272610 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.272719 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.274039 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.274674 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.275912 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.276139 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.276246 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.278702 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:40.278712 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:40.278715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:40.278738 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:40.280082 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.280612 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.280620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.280622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.280696 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.281050 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.281053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.281055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.281241 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:40.281353 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:40.281844 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 07:00:40.281969 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:00:40.282953 7379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.270446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43575 (local address 127.27.110.195:36585)
0812 07:00:40.270934 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:40.270978 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:40.272322 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:40.272897 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:40.272903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:40.273307 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:40.273895 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:40.273910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.274811 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.274814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:40.275777 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:40.275787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.276300 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.276304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:40.276475 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.277022 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:40.277038 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:40.278517 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:40.280213 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:40.280219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:40.280233 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:40.280495 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:40.280795 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:40.280798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:40.280800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:40.280942 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:40.281350 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:40.281358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:40.281610 (+ 252us) client_negotiation.cc:769] Sending connection context
0812 07:00:40.281801 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 07:00:40.282130 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":145,"threads_started":1}
I20260812 07:00:40.284132 7376 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43575
I20260812 07:00:40.284390 7376 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:40.285027 7376 heartbeater.cc:511] Master 127.27.110.254:43575 requested a full tablet report, sending...
I20260812 07:00:40.286054 6575 ts_manager.cc:194] Registered new tserver with Master: e8ff83c3de3a45a4bc732ea7b6f43bf7 (127.27.110.195:39627)
I20260812 07:00:40.286722 6575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:36585
I20260812 07:00:40.287694 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/test-admin.keytab.
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.304852 28091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 07:00:40.307808 7390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:40.306279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39892 (local address 127.27.110.193:32939)
0812 07:00:40.306597 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.306624 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.306647 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:00:40.306697 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.306701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.306875 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.306999 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.307543 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39892: BlockingRecv error: recv got EOF from 127.0.0.1:39892 (error 108)
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":132,"threads_started":1}
W20260812 07:00:40.307804 28091 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc' of type FILE
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.308974 28091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.309859 28091 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:40.319876 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.310751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.27.110.193:32939)
0812 07:00:40.310882 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.310888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.310971 (+ 83us) server_negotiation.cc:407] Connection header received
0812 07:00:40.311131 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.311139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.311178 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.311266 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.311796 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.312598 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.313469 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.313846 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.313966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.316211 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:40.316221 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:40.316232 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:40.316255 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:40.317955 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.318579 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.318584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.318585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.318665 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.318903 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.318905 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.318906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.319226 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:40.319377 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:40.319539 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 07:00:40.319701 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260812 07:00:40.323575 7390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:40.322497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.27.110.193:32939)
0812 07:00:40.322612 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.322616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.322767 (+ 151us) server_negotiation.cc:407] Connection header received
0812 07:00:40.322917 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.322920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.322955 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.323034 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.323471 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39908: BlockingRecv error: recv got EOF from 127.0.0.1:39908 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 07:00:40.323747 28091 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc' of type FILE
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.324671 28091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.325525 28091 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:40.333951 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.326304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.27.110.193:32939)
0812 07:00:40.326421 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.326425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.326528 (+ 103us) server_negotiation.cc:407] Connection header received
0812 07:00:40.326660 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.326662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.326694 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.326776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.327292 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.327788 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.328571 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.328802 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.328906 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.330964 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:40.330967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:40.330968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:40.330989 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:40.332455 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.332878 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.332880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.332881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.332949 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.333173 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.333175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.333176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.333314 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:40.333404 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:40.333669 (+ 265us) server_negotiation.cc:299] Negotiation successful
0812 07:00:40.333783 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260812 07:00:40.336477 7390 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:40.335688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.27.110.193:32939)
0812 07:00:40.335822 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.335826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.335869 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:00:40.335947 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.335949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.335980 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.336036 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.336372 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39922: BlockingRecv error: recv got EOF from 127.0.0.1:39922 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 07:00:40.336549 28091 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786518006756923-28091-0/krb5kdc/krb5cc' of type FILE
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.337491 28091 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:40.338330 28091 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[6503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:00:40.346024 7390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:40.339036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.27.110.193:32939)
0812 07:00:40.339218 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:40.339221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:40.339235 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:40.339337 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:40.339339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:40.339371 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:40.339433 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:40.339855 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.340313 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.341014 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:40.341232 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:40.341302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:40.343421 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:40.343424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:40.343426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:40.343446 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:40.344666 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.345038 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.345040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.345041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.345111 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:40.345329 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:40.345332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:40.345333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:40.345472 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:40.345565 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:40.345759 (+ 194us) server_negotiation.cc:299] Negotiation successful
0812 07:00:40.345875 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 07:00:40.347009 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 6687
I20260812 07:00:40.363989 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 6915
I20260812 07:00:40.386008 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 7155
I20260812 07:00:40.402047 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 6523
2026-08-12T07:00:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2604 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[7411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[7411](info): set up 2 sockets
Aug 12 07:00:40 dist-test-slave-w4v5 krb5kdc[7411](info): commencing operation
krb5kdc: starting...
W20260812 07:00:41.704681 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:41 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab.
Entry for principal oryx/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab.
I20260812 07:00:41.835584 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43403
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab
--principal=oryx/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:41.950053 7431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:41.950340 7431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:41.950423 7431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:41.954437 7431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:41.954528 7431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:41.954566 7431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:41.954603 7431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:41.954639 7431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:41.959860 7431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:43403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab
--principal=oryx/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:41.960904 7431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:41.961954 7431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:41.971253 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:41.971326 7431 server_base.cc:1034] running on GCE node
W20260812 07:00:41.971273 7442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:41.971269 7443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:41.972100 7431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:41.973150 7431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:41.974326 7431 hybrid_clock.cc:648] HybridClock initialized: now 1786518041974297 us; error 39 us; skew 500 ppm
Aug 12 07:00:41 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:41.977289 7431 init.cc:377] Logged in from keytab as oryx/127.27.110.254@KRBTEST.COM (short username oryx)
I20260812 07:00:41.978206 7431 webserver.cc:460] Webserver started at http://127.27.110.254:45455/ using document root <none> and password file <none>
I20260812 07:00:41.978775 7431 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:41.978868 7431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:41.979166 7431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:41.981246 7431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "ca70b32cb9c149819b86023dfb5c2691"
format_stamp: "Formatted at 2026-08-12 07:00:41 on dist-test-slave-w4v5"
server_key: "f69747d531f3f8c2ede7a700e0b71226"
server_key_iv: "c20791e1a2c029023a07e1aa0d147d52"
server_key_version: "encryptionkey@0"
I20260812 07:00:41.981842 7431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "ca70b32cb9c149819b86023dfb5c2691"
format_stamp: "Formatted at 2026-08-12 07:00:41 on dist-test-slave-w4v5"
server_key: "f69747d531f3f8c2ede7a700e0b71226"
server_key_iv: "c20791e1a2c029023a07e1aa0d147d52"
server_key_version: "encryptionkey@0"
I20260812 07:00:41.985617 7431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:00:41.988126 7460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:41.989387 7431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:41.989537 7431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "ca70b32cb9c149819b86023dfb5c2691"
format_stamp: "Formatted at 2026-08-12 07:00:41 on dist-test-slave-w4v5"
server_key: "f69747d531f3f8c2ede7a700e0b71226"
server_key_iv: "c20791e1a2c029023a07e1aa0d147d52"
server_key_version: "encryptionkey@0"
I20260812 07:00:41.989657 7431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:42.004092 7431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:42.006944 7431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:42.007225 7431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:42.014906 7431 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:43403
I20260812 07:00:42.015264 7533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:43403 every 8 connection(s)
I20260812 07:00:42.016007 7431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:42.018826 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:42.022696 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 7431
I20260812 07:00:42.022841 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:42.023180 28091 external_mini_cluster.cc:1428] Setting key dcbd6dff1bd9d2e8c7cd8d2aca9d380c
I20260812 07:00:42.024744 7535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Bootstrap starting.
I20260812 07:00:42.027230 7535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:42.028005 7535 log.cc:824] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:42.030243 7535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: No bootstrap required, opened a new log
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:42.035014 7535 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:00:42.035357 7535 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:42.035413 7535 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca70b32cb9c149819b86023dfb5c2691, State: Initialized, Role: FOLLOWER
I20260812 07:00:42.035955 7535 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:00:42.036154 7535 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:42.036252 7535 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:42.036384 7535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:42.037648 7535 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:00:42.038018 7535 leader_election.cc:302] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca70b32cb9c149819b86023dfb5c2691; no voters:
I20260812 07:00:42.038379 7535 leader_election.cc:288] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:42.038494 7544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:42.038753 7544 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 LEADER]: Becoming Leader. State: Replica: ca70b32cb9c149819b86023dfb5c2691, State: Running, Role: LEADER
I20260812 07:00:42.038852 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.024606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44102 (local address 127.27.110.254:43403)
0812 07:00:42.025045 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:42.025061 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:42.025098 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:00:42.025727 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:42.025744 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:42.026065 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:42.026354 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:42.027373 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.028458 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.029466 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.030144 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.030321 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.033243 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:42.033251 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:42.033270 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:42.033315 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:42.035376 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.035936 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.035942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.035948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.036131 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.036504 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.036507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.036508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.037029 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:42.037243 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:42.037511 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 07:00:42.037845 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":135,"threads_started":1}
I20260812 07:00:42.039342 7544 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:00:42.039680 7535 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:42.041728 7547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } }
I20260812 07:00:42.041859 7547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:42.042133 7546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca70b32cb9c149819b86023dfb5c2691. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } }
I20260812 07:00:42.042254 7546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
W20260812 07:00:42.044968 7573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:42.045053 7573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:42.045135 7572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:42.045804 7572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:42.050730 7572 catalog_manager.cc:1347] Generated new cluster ID: 8fe76e40e29b4ff0950e10be7a8424de
I20260812 07:00:42.050793 7572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:42.065017 7572 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:42.065964 7572 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:42.081513 7572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Generated new TSK 0
I20260812 07:00:42.082324 7572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab.
Entry for principal oryx/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab.
I20260812 07:00:42.118119 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43403
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:42.233958 7581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:42.234280 7581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:42.234359 7581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:42.238343 7581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:42.238445 7581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:42.238539 7581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:42.244238 7581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:42.245311 7581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:42.246330 7581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:42.255892 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.255908 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.255896 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:42.256249 7581 server_base.cc:1034] running on GCE node
I20260812 07:00:42.256714 7581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:42.257314 7581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:42.258528 7581 hybrid_clock.cc:648] HybridClock initialized: now 1786518042258511 us; error 36 us; skew 500 ppm
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:42.261634 7581 init.cc:377] Logged in from keytab as oryx/127.27.110.193@KRBTEST.COM (short username oryx)
I20260812 07:00:42.262596 7581 webserver.cc:460] Webserver started at http://127.27.110.193:41407/ using document root <none> and password file <none>
I20260812 07:00:42.263212 7581 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:42.263317 7581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:42.263537 7581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:42.265575 7581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "676664633f2445849855b82d215f7c7f"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "a2c7e9917e78f8eb60c5d8bc51204727"
server_key_iv: "af5dd0ccee9b32d0a4e29d85b8c53de9"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.266155 7581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "676664633f2445849855b82d215f7c7f"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "a2c7e9917e78f8eb60c5d8bc51204727"
server_key_iv: "af5dd0ccee9b32d0a4e29d85b8c53de9"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.269971 7581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:00:42.272514 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.273737 7581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:42.273885 7581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "676664633f2445849855b82d215f7c7f"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "a2c7e9917e78f8eb60c5d8bc51204727"
server_key_iv: "af5dd0ccee9b32d0a4e29d85b8c53de9"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.274003 7581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:42.286052 7581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:42.289336 7581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:42.289554 7581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:42.290151 7581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:42.290995 7581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:42.291074 7581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.291213 7581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:42.291267 7581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.300205 7581 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:45225
I20260812 07:00:42.300240 7797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:45225 every 8 connection(s)
I20260812 07:00:42.301106 7581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:42.305797 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 7581
I20260812 07:00:42.305938 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:42.306278 28091 external_mini_cluster.cc:1428] Setting key 88edc3bb5452d2c14aeff2967b0a6d0d
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:42.316087 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.302916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:46521 (local address 127.27.110.254:43403)
0812 07:00:42.303060 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:42.303064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:42.303911 (+ 847us) server_negotiation.cc:407] Connection header received
0812 07:00:42.305373 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:42.305376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:42.305428 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:42.305504 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:42.307067 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.307646 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.308832 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.309246 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.309374 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.311947 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:42.311954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:42.311960 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:42.311995 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:42.313783 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.314294 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.314301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.314304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.314412 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.314755 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.314761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.314766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.315027 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:42.315185 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:42.315721 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 07:00:42.315874 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:00:42.316778 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.303298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:46521)
0812 07:00:42.303758 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:42.303802 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:42.305105 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:42.305774 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:42.305785 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:42.306295 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:42.306902 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:42.306917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.307822 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.307825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:42.308700 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:42.308710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.309399 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.309402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:42.309565 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.310195 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:42.310211 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:42.311778 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:42.313907 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:42.313912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:42.313923 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:42.314184 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:42.314517 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:42.314519 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:42.314520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:42.314653 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:42.315189 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:42.315196 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:42.315462 (+ 266us) client_negotiation.cc:769] Sending connection context
0812 07:00:42.315633 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 07:00:42.315880 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for oryx/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:42.318179 7798 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43403
I20260812 07:00:42.318461 7798 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:42.318969 7798 heartbeater.cc:511] Master 127.27.110.254:43403 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:42.320780 7483 ts_manager.cc:194] Registered new tserver with Master: 676664633f2445849855b82d215f7c7f (127.27.110.193:45225)
I20260812 07:00:42.322359 7483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:46521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab.
Entry for principal oryx/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab.
I20260812 07:00:42.336376 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43403
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:42.450059 7808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:42.450328 7808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:42.450371 7808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:42.454521 7808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:42.454583 7808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:42.454644 7808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:42.459833 7808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:42.460758 7808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:42.461690 7808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:42.471302 7816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.471266 7818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.471316 7815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:42.471396 7808 server_base.cc:1034] running on GCE node
I20260812 07:00:42.471951 7808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:42.472631 7808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:42.473831 7808 hybrid_clock.cc:648] HybridClock initialized: now 1786518042473800 us; error 46 us; skew 500 ppm
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:42.476667 7808 init.cc:377] Logged in from keytab as oryx/127.27.110.194@KRBTEST.COM (short username oryx)
I20260812 07:00:42.477645 7808 webserver.cc:460] Webserver started at http://127.27.110.194:43903/ using document root <none> and password file <none>
I20260812 07:00:42.478283 7808 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:42.478379 7808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:42.478624 7808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:42.480759 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "9489275ce6a3415b856fce223d4d97c3"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "b1055c9ef4ad4a97a2b52bf679f97367"
server_key_iv: "4fb514dc73819db7de18d9333f8ea058"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.481348 7808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "9489275ce6a3415b856fce223d4d97c3"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "b1055c9ef4ad4a97a2b52bf679f97367"
server_key_iv: "4fb514dc73819db7de18d9333f8ea058"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.485237 7808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260812 07:00:42.487650 7829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.488973 7808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:42.489125 7808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "9489275ce6a3415b856fce223d4d97c3"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "b1055c9ef4ad4a97a2b52bf679f97367"
server_key_iv: "4fb514dc73819db7de18d9333f8ea058"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.489240 7808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:42.509650 7808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:42.512571 7808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:42.512811 7808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:42.513397 7808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:42.514211 7808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:42.514282 7808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.514355 7808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:42.514408 7808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.523557 7808 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:44893
I20260812 07:00:42.523631 8015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:44893 every 8 connection(s)
I20260812 07:00:42.524472 7808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:42.533226 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 7808
I20260812 07:00:42.533325 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:42.533555 28091 external_mini_cluster.cc:1428] Setting key 9b2f76b4de8760bd889f01dc53d3594d
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:42.539261 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.526300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:42401 (local address 127.27.110.254:43403)
0812 07:00:42.526435 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:42.526439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:42.527260 (+ 821us) server_negotiation.cc:407] Connection header received
0812 07:00:42.528750 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:42.528754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:42.528815 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:42.528919 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:42.530199 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.530738 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.531973 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.532210 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.532372 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.535337 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:42.535344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:42.535348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:42.535378 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:42.537044 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.537591 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.537600 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.537601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.537682 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.538097 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.538100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.538104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.538296 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:42.538415 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:42.538935 (+ 520us) server_negotiation.cc:299] Negotiation successful
0812 07:00:42.539069 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:00:42.540174 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.526600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.194:42401)
0812 07:00:42.527048 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:42.527145 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:42.528530 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:42.529114 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:42.529121 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:42.529470 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:42.530015 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:42.530030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.530868 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.530872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:42.531844 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:42.531854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.532421 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.532425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:42.532595 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.533538 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:42.533555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:42.535174 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:42.537179 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:42.537184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:42.537196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:42.537464 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:42.537783 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:42.537786 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:42.537788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:42.537933 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:42.538412 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:42.538419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:42.538681 (+ 262us) client_negotiation.cc:769] Sending connection context
0812 07:00:42.538872 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 07:00:42.539260 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260812 07:00:42.541347 8016 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43403
I20260812 07:00:42.541611 8016 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:42.542169 8016 heartbeater.cc:511] Master 127.27.110.254:43403 requested a full tablet report, sending...
I20260812 07:00:42.543390 7483 ts_manager.cc:194] Registered new tserver with Master: 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893)
I20260812 07:00:42.544220 7483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:42401
WARNING: no policy specified for oryx/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab.
Entry for principal oryx/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab.
I20260812 07:00:42.565900 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43403
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:42.682535 8028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:42.682787 8028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:42.682832 8028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:42.686836 8028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:42.686896 8028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:42.686956 8028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:42.692144 8028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:43403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:42.693220 8028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:42.694327 8028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:42.703742 8037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.703765 8039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.703750 8041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:42.703984 8028 server_base.cc:1034] running on GCE node
I20260812 07:00:42.704455 8028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:42.705049 8028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:42.706215 8028 hybrid_clock.cc:648] HybridClock initialized: now 1786518042706195 us; error 32 us; skew 500 ppm
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:42.709028 8028 init.cc:377] Logged in from keytab as oryx/127.27.110.195@KRBTEST.COM (short username oryx)
I20260812 07:00:42.709890 8028 webserver.cc:460] Webserver started at http://127.27.110.195:46317/ using document root <none> and password file <none>
I20260812 07:00:42.710413 8028 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:42.710489 8028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:42.710664 8028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:42.712792 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "2c6246dc472d4d4b9f04d5b72d726c90"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "6c677c6e070633c17f0857c0b92c069b"
server_key_iv: "abf403f24c1fbdc19afbf2d8796b2f05"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.713311 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "2c6246dc472d4d4b9f04d5b72d726c90"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "6c677c6e070633c17f0857c0b92c069b"
server_key_iv: "abf403f24c1fbdc19afbf2d8796b2f05"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.717186 8028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:42.719857 8054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.721192 8028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:42.721295 8028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "2c6246dc472d4d4b9f04d5b72d726c90"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "6c677c6e070633c17f0857c0b92c069b"
server_key_iv: "abf403f24c1fbdc19afbf2d8796b2f05"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.721388 8028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:42.738495 8028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:42.741860 8028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:42.742019 8028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:42.742555 8028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:42.743371 8028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:42.743422 8028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.743462 8028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:42.743505 8028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.752871 8028 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:41819
I20260812 07:00:42.752907 8255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:41819 every 8 connection(s)
I20260812 07:00:42.753796 8028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:42.763373 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 8028
I20260812 07:00:42.763913 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:42.764355 28091 external_mini_cluster.cc:1428] Setting key 464d56442d2c19eb55227dea93062cb1
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:42.769922 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.755659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:46229 (local address 127.27.110.254:43403)
0812 07:00:42.755816 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:42.755821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:42.756805 (+ 984us) server_negotiation.cc:407] Connection header received
0812 07:00:42.758679 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:42.758683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:42.758772 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:42.758900 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:42.760310 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.760851 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.762535 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.762909 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.763017 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.766378 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:42.766381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:42.766384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:42.766411 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:42.768101 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.768592 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.768595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.768596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.768674 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:42.768979 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:42.768981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:42.768982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:42.769172 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:42.769248 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:42.769706 (+ 458us) server_negotiation.cc:299] Negotiation successful
0812 07:00:42.769804 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:00:42.770831 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.756095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:46229)
0812 07:00:42.756585 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:42.756645 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:42.758394 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:42.759148 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:42.759157 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:42.759535 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:42.760153 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:42.760168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.760988 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.760991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:42.762404 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:42.762415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:42.763063 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:42.763066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:42.763276 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:42.763912 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:42.763933 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:42.766171 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:42.768222 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:42.768227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:42.768239 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:42.768503 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:42.768756 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:42.768758 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:42.768760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:42.768898 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:42.769265 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:42.769270 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:42.769500 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 07:00:42.769659 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 07:00:42.769916 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260812 07:00:42.771957 8258 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43403
I20260812 07:00:42.772214 8258 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:42.772718 8258 heartbeater.cc:511] Master 127.27.110.254:43403 requested a full tablet report, sending...
I20260812 07:00:42.773738 7483 ts_manager.cc:194] Registered new tserver with Master: 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819)
I20260812 07:00:42.774312 7483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:46229
I20260812 07:00:42.779585 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.253.keytab.
Entry for principal oryx/127.27.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.253.keytab.
I20260812 07:00:42.810593 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:37331
--webserver_interface=127.27.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.253.keytab
--principal=oryx/127.27.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:43403,127.27.110.253:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:42.928117 8267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:42.928452 8267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:42.928565 8267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:42.932605 8267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:42.932682 8267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:42.932720 8267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:42.932776 8267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:42.932813 8267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:42.938156 8267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:43403,127.27.110.253:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.253.keytab
--principal=oryx/127.27.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
W20260812 07:00:42.938424 8267 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.110.254:43403,127.27.110.253:37331'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 07:00:42.939355 8267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:42.940465 8267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:42.949112 8278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.949128 8284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:42.949120 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:42.949581 8267 server_base.cc:1034] running on GCE node
I20260812 07:00:42.950207 8267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:42.951388 8267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:42.952600 8267 hybrid_clock.cc:648] HybridClock initialized: now 1786518042952572 us; error 52 us; skew 500 ppm
Aug 12 07:00:42 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:42.955480 8267 init.cc:377] Logged in from keytab as oryx/127.27.110.253@KRBTEST.COM (short username oryx)
I20260812 07:00:42.956295 8267 webserver.cc:460] Webserver started at http://127.27.110.253:34743/ using document root <none> and password file <none>
I20260812 07:00:42.956799 8267 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:42.956857 8267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:42.957020 8267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:42.959338 8267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/instance:
uuid: "0d95f9f8d7ea48519c09223c32d7337a"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "2afd2b3fd6faf7ebcfa894da4a38004a"
server_key_iv: "b2dd3e9c4230a4722dcc0b6cb551ac32"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.959867 8267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal/instance:
uuid: "0d95f9f8d7ea48519c09223c32d7337a"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "2afd2b3fd6faf7ebcfa894da4a38004a"
server_key_iv: "b2dd3e9c4230a4722dcc0b6cb551ac32"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.963594 8267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:00:42.965943 8294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:42.967257 8267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:42.967394 8267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
uuid: "0d95f9f8d7ea48519c09223c32d7337a"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "2afd2b3fd6faf7ebcfa894da4a38004a"
server_key_iv: "b2dd3e9c4230a4722dcc0b6cb551ac32"
server_key_version: "encryptionkey@0"
I20260812 07:00:42.967509 8267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:42.982060 8267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:42.985395 8267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:42.985612 8267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:42.992944 8267 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.253:37331
I20260812 07:00:42.993369 8379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.253:37331 every 8 connection(s)
I20260812 07:00:42.994131 8267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
I20260812 07:00:42.997012 8381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:42.998230 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 8267
I20260812 07:00:43.001317 8381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.253@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:43.013085 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.003465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.27.110.254:43403)
0812 07:00:43.003586 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.003590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.003895 (+ 305us) server_negotiation.cc:407] Connection header received
0812 07:00:43.004082 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.004088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.004147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.004241 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.004957 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.005487 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.006422 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.006714 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.006820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.009553 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.009556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.009558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.009609 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.011221 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.011702 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.011707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.011709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.011797 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.012098 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.012102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.012103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.012329 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.012441 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.012799 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.012926 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 07:00:43.013787 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.003288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:44110)
0812 07:00:43.003763 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.003801 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.003963 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.004293 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.004299 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.004704 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.004824 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.004830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.005638 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.005641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.006293 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.006302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.006895 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.006902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.007049 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.007643 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.007660 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.009397 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.011335 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.011347 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.011352 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.011594 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.011893 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.011897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.011899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.011991 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.012441 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.012449 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.012570 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.012729 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.012942 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":22,"thread_start_us":101,"threads_started":1}
I20260812 07:00:43.013795 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:42.999789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.27.110.253:37331)
0812 07:00:43.000223 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.000250 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.000287 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:00:43.000940 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.000957 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.001285 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.001565 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.002517 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.003966 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.004997 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.005489 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.005670 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.008677 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.008683 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.008716 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.008738 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.010562 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.011039 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.011050 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.011054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.011225 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.011528 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.011531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.011533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.011985 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.012216 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.012546 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.012892 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"spinlock_wait_cycles":7424,"thread_start_us":148,"threads_started":1}
I20260812 07:00:43.015585 8381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518042, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.253@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
I20260812 07:00:43.025151 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.016116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:32946)
0812 07:00:43.016238 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.016254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.016328 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.016840 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.016842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.016972 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.017061 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.017066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.018537 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.018539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.019043 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.019050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.019662 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.019665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.019849 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.020321 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.020333 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.021820 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.023697 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.023700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.023701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.023891 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.024231 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.024233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.024235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.024324 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.024760 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.024763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.024843 (+ 80us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.024934 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.025035 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":71}
I20260812 07:00:43.025279 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.016504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.27.110.253:37331)
0812 07:00:43.016636 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.016640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.016653 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:00:43.016685 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.016688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.016727 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.016809 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.017170 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.018429 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.019182 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.019556 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.019654 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.021950 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.021954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.021957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.021980 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.023500 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.023996 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.023999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.024001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.024090 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.024422 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.024425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.024427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.024639 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.024730 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.024999 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.025105 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":59}
I20260812 07:00:43.028997 8381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Bootstrap starting.
I20260812 07:00:43.031476 8381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:43.032167 8381 log.cc:824] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:43.034119 8381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: No bootstrap required, opened a new log
I20260812 07:00:43.037236 8381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } }
I20260812 07:00:43.037442 8381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:43.037477 8381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d95f9f8d7ea48519c09223c32d7337a, State: Initialized, Role: FOLLOWER
I20260812 07:00:43.037875 8381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } }
I20260812 07:00:43.038308 8390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } } }
I20260812 07:00:43.038450 8390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:43.039004 8381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:00:43.044663 8418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:43.044756 8418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:43.046772 8267 master_runner.cc:428] Detected that this master 0d95f9f8d7ea48519c09223c32d7337a is joining an existing cluster
I20260812 07:00:43.046824 8267 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.110.253:37331
I20260812 07:00:43.057914 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.049253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:32956)
0812 07:00:43.049687 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.049700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.049815 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.050121 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.050124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.050349 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.050495 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.050504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.051464 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.051467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.052076 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.052084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.052660 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.052664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.052759 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.053739 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.053758 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.054149 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.056241 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.056249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.056499 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.056795 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.056800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.056886 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.057310 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.057313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.057399 (+ 86us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.057553 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.057716 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":93,"threads_started":1}
I20260812 07:00:43.058022 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.048828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:44112)
0812 07:00:43.049526 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.049539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.049661 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.049906 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.049909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.050156 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.050279 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.050285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.051427 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.051431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.052536 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.052548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.053282 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.053285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.053378 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.053852 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.053868 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.054349 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.056300 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.056305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.056627 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.056948 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.056955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.057048 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.057481 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.057487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.057579 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.057716 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.057831 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":70,"thread_start_us":94,"threads_started":1}
I20260812 07:00:43.058631 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.048919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.27.110.254:43403)
0812 07:00:43.049043 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.049048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.049623 (+ 575us) server_negotiation.cc:407] Connection header received
0812 07:00:43.049745 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.049748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.049794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.049890 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.050408 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.051302 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.052681 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.053031 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.053114 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.054496 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.054503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.054507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.054538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.056188 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056746 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.056752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.056757 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.056846 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.057155 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.057159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.057161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.057378 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.057479 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.057779 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.057897 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:00:43.058991 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.049393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32956 (local address 127.27.110.253:37331)
0812 07:00:43.049497 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.049502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.049762 (+ 260us) server_negotiation.cc:407] Connection header received
0812 07:00:43.049904 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.049907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.049941 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.049984 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.050623 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.051377 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.052203 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.052544 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.052659 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.054290 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.054293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.054296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.054353 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.056103 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056600 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.056604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.056606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.056719 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.056995 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.057000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.057002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.057218 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.057333 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.057619 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.058845 (+ 1226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1117,"server-negotiator.queue_time_us":34,"spinlock_wait_cycles":59648}
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.254@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
I20260812 07:00:43.073308 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.063178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.27.110.253:37331)
0812 07:00:43.063289 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.063293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.064046 (+ 753us) server_negotiation.cc:407] Connection header received
0812 07:00:43.064236 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.064238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.064281 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.064347 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.065104 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.065568 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.066793 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.067082 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.067203 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.069611 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.069616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.069624 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.069645 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.071488 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.072053 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.072056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.072058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.072156 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.072473 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.072475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.072476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.072634 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.072731 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.073111 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.073201 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:00:43.073652 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.063118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:32966)
0812 07:00:43.063916 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.063968 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.064103 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.064407 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.064413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.064856 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.064994 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.065001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.065678 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.065683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.066666 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.066677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.067210 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.067215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.067332 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.067954 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.067971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.069469 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.071610 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.071619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.071626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.071927 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.072260 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.072266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.072270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.072376 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.072743 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.072752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.072897 (+ 145us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.073063 (+ 166us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.073287 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":92,"threads_started":1}
I20260812 07:00:43.075220 7483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.110.253:37331: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a"
member_type: NON_VOTER
last_known_addr {
host: "127.27.110.253"
port: 37331
}
attrs {
promote: true
}
}
dest_uuid: "ca70b32cb9c149819b86023dfb5c2691"
cas_config_opid_index: -1
I20260812 07:00:43.075924 7483 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } }
I20260812 07:00:43.077719 8433 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0d95f9f8d7ea48519c09223c32d7337a (127.27.110.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } }
I20260812 07:00:43.078647 8340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 0 FOLLOWER]: Advancing to term 1
WARNING: no policy specified for oryx/127.27.110.252@KRBTEST.COM; defaulting to no policy
I20260812 07:00:43.080055 8433 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.110.253:37331
I20260812 07:00:43.080144 8340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Refusing update from remote peer ca70b32cb9c149819b86023dfb5c2691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 07:00:43.080624 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } }
I20260812 07:00:43.080691 8267 master.cc:537] Master@127.27.110.253:37331 shutting down...
I20260812 07:00:43.081111 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:43.080799 8433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.252@KRBTEST.COM" created.
W20260812 07:00:43.083523 7467 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260812 07:00:43.084548 7467 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 -> Peer 0d95f9f8d7ea48519c09223c32d7337a (127.27.110.253:37331): Couldn't send request to peer 0d95f9f8d7ea48519c09223c32d7337a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 07:00:43.086767 8267 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:43.087116 8267 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:43.087241 8267 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: stopping tablet replica
I20260812 07:00:43.091701 8267 master.cc:552] Master@127.27.110.253:37331 shutdown complete.
I20260812 07:00:43.091773 8267 master_runner.cc:305] Clearing existing system tablet
I20260812 07:00:43.092013 8267 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.092085 8267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.093391 8267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 07:00:43.095364 8439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.095635 8267 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.095763 8267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:43.095804 8267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
uuid: "0d95f9f8d7ea48519c09223c32d7337a"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "2afd2b3fd6faf7ebcfa894da4a38004a"
server_key_iv: "b2dd3e9c4230a4722dcc0b6cb551ac32"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.098392 8267 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:43.100570 8267 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 07:00:43.100682 8267 log.cc:1196] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.252.keytab.
Entry for principal oryx/127.27.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.252.keytab.
I20260812 07:00:43.100981 8267 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Deleting consensus metadata
I20260812 07:00:43.101169 8267 master_runner.cc:315] Copying system tablet from 127.27.110.254:43403
I20260812 07:00:43.101999 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:42849
--webserver_interface=127.27.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.252.keytab
--principal=oryx/127.27.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:00:43.102677 8267 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.110.254:43403
I20260812 07:00:43.112906 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.103484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44120 (local address 127.27.110.254:43403)
0812 07:00:43.103622 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.103626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.104099 (+ 473us) server_negotiation.cc:407] Connection header received
0812 07:00:43.104320 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.104325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.104395 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.104507 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.105032 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.105773 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.106973 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.107341 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.107420 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.109035 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.109040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.109042 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.109069 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.110796 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.111476 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.111480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.111482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.111576 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.111922 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.111925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.111927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.112145 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.112244 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.112625 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.112743 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:00:43.113026 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.103722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:44120)
0812 07:00:43.104007 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.104021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.104181 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.104541 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.104544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.104761 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.104899 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.104906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.105892 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.105896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.106831 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.106842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.107595 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.107599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.107688 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.108416 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.108431 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.108876 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.110915 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.110922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.110926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.111355 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.111690 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.111694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.111698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.111812 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.112261 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.112266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.112399 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.112545 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.112729 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260812 07:00:43.113917 7516 tablet_copy_service.cc:140] P ca70b32cb9c149819b86023dfb5c2691: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d95f9f8d7ea48519c09223c32d7337a ({username='oryx', principal='oryx/127.27.110.253@KRBTEST.COM'} at 127.0.0.1:44120)
I20260812 07:00:43.114142 7516 tablet_copy_service.cc:161] P ca70b32cb9c149819b86023dfb5c2691: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d95f9f8d7ea48519c09223c32d7337a at {username='oryx', principal='oryx/127.27.110.253@KRBTEST.COM'} at 127.0.0.1:44120: session id = 0d95f9f8d7ea48519c09223c32d7337a-00000000000000000000000000000000
I20260812 07:00:43.116487 7516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 07:00:43.118391 8267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 07:00:43.120798 8267 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: tablet copy: Starting download of 0 data blocks...
I20260812 07:00:43.120998 8267 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: tablet copy: Starting download of 1 WAL segments...
I20260812 07:00:43.122836 8267 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 07:00:43.124686 7516 tablet_copy_service.cc:342] P ca70b32cb9c149819b86023dfb5c2691: Request end of tablet copy session 0d95f9f8d7ea48519c09223c32d7337a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.110.253@KRBTEST.COM'} at 127.0.0.1:44120
I20260812 07:00:43.124876 7516 tablet_copy_service.cc:433] P ca70b32cb9c149819b86023dfb5c2691: ending tablet copy session 0d95f9f8d7ea48519c09223c32d7337a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d95f9f8d7ea48519c09223c32d7337a
I20260812 07:00:43.132869 8267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:43.134577 8449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.135946 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.136268 8267 server_base.cc:1034] running on GCE node
W20260812 07:00:43.136385 8456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.136610 8267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:43.136895 8267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:43.138013 8267 hybrid_clock.cc:648] HybridClock initialized: now 1786518043137998 us; error 32 us; skew 500 ppm
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:43.139714 8267 init.cc:377] Logged in from keytab as oryx/127.27.110.253@KRBTEST.COM (short username oryx)
I20260812 07:00:43.140152 8267 webserver.cc:460] Webserver started at http://127.27.110.253:38407/ using document root <none> and password file <none>
I20260812 07:00:43.140326 8267 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.140359 8267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.141629 8267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:43.142665 8471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.142910 8267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:43.142980 8267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
uuid: "0d95f9f8d7ea48519c09223c32d7337a"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "2afd2b3fd6faf7ebcfa894da4a38004a"
server_key_iv: "b2dd3e9c4230a4722dcc0b6cb551ac32"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.143055 8267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.154999 8267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:43.157123 8267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:43.161981 8267 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.253:37331
I20260812 07:00:43.162242 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.253:37331 every 8 connection(s)
I20260812 07:00:43.162725 8267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
I20260812 07:00:43.164738 8557 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:00:43.165723 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Bootstrap starting.
I20260812 07:00:43.176332 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:00:43.176641 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: Bootstrap complete.
I20260812 07:00:43.177219 8557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } }
I20260812 07:00:43.177359 8557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d95f9f8d7ea48519c09223c32d7337a, State: Initialized, Role: LEARNER
I20260812 07:00:43.177469 8557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } }
I20260812 07:00:43.177788 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } }
I20260812 07:00:43.177911 8557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:43.178032 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: LEARNER
I20260812 07:00:43.181217 8581 catalog_manager.cc:1259] Loaded cluster ID: 8fe76e40e29b4ff0950e10be7a8424de
I20260812 07:00:43.181326 8581 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: loading cluster ID for follower catalog manager: success
I20260812 07:00:43.183552 8581 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: acquiring CA information for follower catalog manager: success
I20260812 07:00:43.184960 8581 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 07:00:43.237300 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:43.237632 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:43.237744 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:43.242226 8447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:43.242311 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:43.242348 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:43.242408 8447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:43.242445 8447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:43.247917 8447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.252.keytab
--principal=oryx/127.27.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:43.249004 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:43.250064 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:43.258782 8588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.258792 8592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.258792 8595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.259011 8447 server_base.cc:1034] running on GCE node
I20260812 07:00:43.259680 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:43.260803 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:43.262005 8447 hybrid_clock.cc:648] HybridClock initialized: now 1786518043261956 us; error 47 us; skew 500 ppm
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:43.264998 8447 init.cc:377] Logged in from keytab as oryx/127.27.110.252@KRBTEST.COM (short username oryx)
I20260812 07:00:43.265830 8447 webserver.cc:460] Webserver started at http://127.27.110.252:41071/ using document root <none> and password file <none>
I20260812 07:00:43.266353 8447 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.266413 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.266580 8447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:43.268685 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/instance:
uuid: "83e4b9bd53a44347828fd9b9eb3df233"
format_stamp: "Formatted at 2026-08-12 07:00:43 on dist-test-slave-w4v5"
server_key: "a880702f05a9a5a99c08c6447d82b932"
server_key_iv: "bf535b1ccfbe310cfb12b1d4b9d02ca0"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.269207 8447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal/instance:
uuid: "83e4b9bd53a44347828fd9b9eb3df233"
format_stamp: "Formatted at 2026-08-12 07:00:43 on dist-test-slave-w4v5"
server_key: "a880702f05a9a5a99c08c6447d82b932"
server_key_iv: "bf535b1ccfbe310cfb12b1d4b9d02ca0"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.273072 8447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:43.275609 8606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.276913 8447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:43.277118 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
uuid: "83e4b9bd53a44347828fd9b9eb3df233"
format_stamp: "Formatted at 2026-08-12 07:00:43 on dist-test-slave-w4v5"
server_key: "a880702f05a9a5a99c08c6447d82b932"
server_key_iv: "bf535b1ccfbe310cfb12b1d4b9d02ca0"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.277228 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.301287 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:43.304669 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:43.304879 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:43.311983 8447 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.252:42849
I20260812 07:00:43.312278 8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.252:42849 every 8 connection(s)
I20260812 07:00:43.313040 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
I20260812 07:00:43.315869 8698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:43.319626 8698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:00:43.320883 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 8447
I20260812 07:00:43.324687 7798 heartbeater.cc:503] Master 127.27.110.254:43403 was elected leader, sending a full tablet report...
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.252@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.110.252@KRBTEST.COM
I20260812 07:00:43.334589 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.321617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.27.110.254:43403)
0812 07:00:43.321761 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.321766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.322725 (+ 959us) server_negotiation.cc:407] Connection header received
0812 07:00:43.323662 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.323666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.323726 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.323812 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.324868 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.325684 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.327274 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.327495 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.327580 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.330466 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.330470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.330473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.330503 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.332314 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.333002 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.333005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.333007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.333089 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.333470 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.333473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.333475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.333663 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.333773 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.334265 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.334384 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:00:43.335443 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.321997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:44122)
0812 07:00:43.322585 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.322625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.323424 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.323890 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.323895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.324325 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.324685 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.324699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.325850 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.325853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.327165 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.327176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.327640 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.327643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.327793 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.328438 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.328514 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.330278 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.332428 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.332435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.332445 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.332885 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.333205 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.333209 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.333211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.333354 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.333789 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.333797 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.334042 (+ 245us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.334222 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.334528 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":150,"threads_started":1}
I20260812 07:00:43.335440 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.321997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50558 (local address 127.27.110.252:42849)
0812 07:00:43.322552 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.322570 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.322608 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:00:43.323419 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.323438 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.323633 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.323814 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.324685 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.325985 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.327677 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.328054 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.328166 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.331018 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.331027 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.331042 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.331064 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.333029 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.333452 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.333458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.333460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.333548 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.333826 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.333829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.333831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.334087 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.334328 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.334495 (+ 167us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.334638 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"spinlock_wait_cycles":29952,"thread_start_us":165,"threads_started":1}
I20260812 07:00:43.336776 8698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.252@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
I20260812 07:00:43.345492 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.337301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:32970)
0812 07:00:43.337501 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.337516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.337591 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.337978 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.337981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.338141 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.338233 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.338238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.339184 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.339188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.339784 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.339793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.340295 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.340298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.340397 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.340886 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.340898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.342205 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.344037 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.344040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.344041 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.344248 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.344635 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.344637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.344638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.344720 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.345084 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.345086 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.345164 (+ 78us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.345259 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.345360 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 07:00:43.345600 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.337421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.27.110.253:37331)
0812 07:00:43.337773 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.337776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.337788 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:00:43.337822 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.337824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.337863 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.337971 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.338357 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.339028 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.339910 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.340174 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.340285 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.342360 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.342363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.342365 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.342388 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.343933 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.344434 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.344443 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.344444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.344533 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.344810 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.344812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.344813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.344992 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.345076 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.345326 (+ 250us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.345432 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":102,"threads_started":1}
I20260812 07:00:43.346331 8698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.252@KRBTEST.COM for oryx/127.27.110.252@KRBTEST.COM
I20260812 07:00:43.354293 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.346895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.27.110.252:42849)
0812 07:00:43.347012 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.347016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.347040 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:00:43.347184 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.347187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.347235 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.347320 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.347706 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.348232 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.349005 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.349264 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.349378 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.351349 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.351353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.351356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.351387 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.352636 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.353067 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.353072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.353074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.353154 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.353447 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.353450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.353452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.353639 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.353750 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.354014 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.354157 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":53}
I20260812 07:00:43.354292 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.346806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:50570)
0812 07:00:43.346912 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.346948 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.347029 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.347332 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.347336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.347499 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.347561 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.347566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.348347 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.348352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.348879 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.348885 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.349380 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.349384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.349465 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.349827 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.349839 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.351178 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.352752 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.352759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.352761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.352961 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.353259 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.353262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.353264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.353346 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.353750 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.353753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.353845 (+ 92us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.353952 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.354099 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 07:00:43.357923 8698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Bootstrap starting.
I20260812 07:00:43.360498 8698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:43.361245 8698 log.cc:824] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:43.363247 8698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: No bootstrap required, opened a new log
I20260812 07:00:43.366361 8698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } }
I20260812 07:00:43.366560 8698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:43.366596 8698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83e4b9bd53a44347828fd9b9eb3df233, State: Initialized, Role: FOLLOWER
I20260812 07:00:43.367040 8698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } }
I20260812 07:00:43.367558 8708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } } }
I20260812 07:00:43.367679 8708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:43.368299 8698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:00:43.373211 8726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:43.373315 8726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:43.376956 8447 master_runner.cc:428] Detected that this master 83e4b9bd53a44347828fd9b9eb3df233 is joining an existing cluster
I20260812 07:00:43.377029 8447 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.110.252:42849
I20260812 07:00:43.388154 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.379901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.27.110.252:42849)
0812 07:00:43.380032 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.380036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.380051 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:00:43.380092 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.380095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.380141 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.380233 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.380798 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.381572 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.382694 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.383005 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.383107 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.384889 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.384893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.384896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.384926 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.386483 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.386922 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.386927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.386928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.387013 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.387319 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.387324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.387326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.387494 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.387605 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.387868 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.388018 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":896}
I20260812 07:00:43.388166 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.379440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:50582)
0812 07:00:43.379829 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.379846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.379940 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.380255 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.380262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.380514 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.380684 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.380693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.381678 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.381680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.382586 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.382597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.383266 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.383269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.383375 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.383936 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.383951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.384463 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.386615 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.386618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.386620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.386820 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.387131 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.387134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.387135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.387228 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.387593 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.387596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.387710 (+ 114us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.387821 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.387961 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":3,"thread_start_us":146,"threads_started":1}
I20260812 07:00:43.388914 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 7581
I20260812 07:00:43.417950 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.379364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.27.110.254:43403)
0812 07:00:43.379476 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.379481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.380582 (+ 1101us) server_negotiation.cc:407] Connection header received
0812 07:00:43.380783 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.380788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.380840 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.380898 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.381586 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.382436 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.383663 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.384054 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.384142 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.414238 (+ 30096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.414242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.414245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.414280 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.416045 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.416551 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.416554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.416557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.416640 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.416932 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.416935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.416937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.417186 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.417258 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.417640 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.417774 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 07:00:43.418556 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.379525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:44126)
0812 07:00:43.380494 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.380507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.380649 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.381078 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.381081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.381335 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.381454 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.381461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.382609 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.382613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.383521 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.383534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.384304 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.384307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.384422 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.384894 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.384910 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.413344 (+ 28434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.416185 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.416188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.416190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.416437 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.416749 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.416751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.416753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.416836 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.417337 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.417340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.417450 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.418203 (+ 753us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.418370 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260812 07:00:43.418926 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.380020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32984 (local address 127.27.110.253:37331)
0812 07:00:43.380138 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.380144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.380654 (+ 510us) server_negotiation.cc:407] Connection header received
0812 07:00:43.380941 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.380945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.381001 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.381079 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.381995 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.382904 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.384334 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.384734 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.384817 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.414238 (+ 29421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.414245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.414251 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.414295 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.416775 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.417348 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.417355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.417359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.417453 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.417821 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.417827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.417832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.418090 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.418170 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.418569 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.418734 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 07:00:43.419612 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.380166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:32984)
0812 07:00:43.380553 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.380570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.380778 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.381254 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.381257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.381521 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.381698 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.381707 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.383169 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.383174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.384178 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.384190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.384989 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.384993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.385093 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.413562 (+ 28469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.413580 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.414002 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.416930 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.416936 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.416941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.417220 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.417582 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.417588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.417593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.417691 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.418257 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.418260 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.418373 (+ 113us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.419154 (+ 781us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.419336 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260812 07:00:43.422279 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:45225
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:43403
--webserver_port=41407
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518041, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.254@KRBTEST.COM for oryx/127.27.110.252@KRBTEST.COM
I20260812 07:00:43.431850 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.423281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.27.110.252:42849)
0812 07:00:43.423403 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.423406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.423417 (+ 11us) server_negotiation.cc:407] Connection header received
0812 07:00:43.423473 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.423475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.423515 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.423611 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.424133 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.424927 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.426113 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.426325 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.426448 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.428582 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.428592 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.428595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.428616 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.430108 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.430560 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.430564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.430566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.430647 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.430940 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.430943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.430945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.431199 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.431304 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.431585 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.431702 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:00:43.431850 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.422810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:50592)
0812 07:00:43.423006 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.423028 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.423139 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.423630 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.423633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.423824 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.423978 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.423987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.425079 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.425082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.425989 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.426000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.426453 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.426457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.426559 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.427019 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.427031 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.428444 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.430234 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.430237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.430239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.430443 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.430749 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.430752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.430754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.430844 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.431309 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.431315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.431417 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.431528 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.431682 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 07:00:43.433156 7483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.110.252:42849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233"
member_type: NON_VOTER
last_known_addr {
host: "127.27.110.252"
port: 42849
}
attrs {
promote: true
}
}
dest_uuid: "ca70b32cb9c149819b86023dfb5c2691"
cas_config_opid_index: 5
I20260812 07:00:43.433766 7483 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } }
I20260812 07:00:43.435901 8650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:43.437134 8650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Refusing update from remote peer ca70b32cb9c149819b86023dfb5c2691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 07:00:43.434917 8433 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.438885 8433 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.110.252:42849
I20260812 07:00:43.439167 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.439282 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:43.439392 8447 master.cc:537] Master@127.27.110.252:42849 shutting down...
I20260812 07:00:43.439391 8433 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 07:00:43.441347 7461 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 -> Peer 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252:42849): Couldn't send request to peer 83e4b9bd53a44347828fd9b9eb3df233. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 07:00:43.446489 8447 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:43.446570 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.435046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:32996)
0812 07:00:43.442339 (+ 7293us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.442358 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.442452 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.442771 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.442775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.442792 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:43.442891 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.442900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.443982 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.443989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.445235 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.445251 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.446128 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.446132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.446218 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.446262 (+ 44us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.446347 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.446436 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6803}
I20260812 07:00:43.446798 8447 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:43.446910 8447 tablet_replica.cc:330] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: stopping tablet replica
I20260812 07:00:43.446924 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.435363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.27.110.253:37331)
0812 07:00:43.435488 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.435493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.442483 (+ 6990us) server_negotiation.cc:407] Connection header received
0812 07:00:43.442551 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.442555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.442625 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.442747 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:43.443033 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.443834 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.445353 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.445992 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.446128 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.446286 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.446727 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.446811 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:43.447888 8515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 LEARNER]: Refusing update from remote peer ca70b32cb9c149819b86023dfb5c2691: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 07:00:43.448361 8740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 07:00:43.450412 8515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.453106 8433 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: attempting to promote NON_VOTER 0d95f9f8d7ea48519c09223c32d7337a to VOTER
I20260812 07:00:43.453056 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca70b32cb9c149819b86023dfb5c2691. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.453173 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: LEARNER
I20260812 07:00:43.453264 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: NON_VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: true } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.453326 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: LEARNER
I20260812 07:00:43.453632 8433 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } }
I20260812 07:00:43.455247 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.454854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:50608)
0812 07:00:43.455055 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.455165 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:42849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260812 07:00:43.455525 8515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 LEARNER]: Refusing update from remote peer ca70b32cb9c149819b86023dfb5c2691: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 07:00:43.455921 8740 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 07:00:43.456394 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.456143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:50620)
0812 07:00:43.456273 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.456330 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:42849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 07:00:43.457026 7461 consensus_peers.cc:487] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 -> Peer 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252:42849): Couldn't send request to peer 83e4b9bd53a44347828fd9b9eb3df233. Status: Network error: Client connection negotiation failed: client connection to 127.27.110.252:42849: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 07:00:43.457679 8434 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0d95f9f8d7ea48519c09223c32d7337a (127.27.110.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.458252 8515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0d95f9f8d7ea48519c09223c32d7337a (127.27.110.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.459457 8434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.459421 8433 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.459537 8434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:43.459544 8433 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:43.460552 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.460680 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:43.465932 8447 master.cc:552] Master@127.27.110.252:42849 shutdown complete.
I20260812 07:00:43.466012 8447 master_runner.cc:305] Clearing existing system tablet
I20260812 07:00:43.466259 8447 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.466333 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.467481 8447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260812 07:00:43.468374 8746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.468874 8447 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.469029 8447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:43.469103 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
uuid: "83e4b9bd53a44347828fd9b9eb3df233"
format_stamp: "Formatted at 2026-08-12 07:00:43 on dist-test-slave-w4v5"
server_key: "a880702f05a9a5a99c08c6447d82b932"
server_key_iv: "bf535b1ccfbe310cfb12b1d4b9d02ca0"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.471204 8447 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:43.473068 8447 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 07:00:43.473182 8447 log.cc:1196] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 07:00:43.473457 8447 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Deleting consensus metadata
I20260812 07:00:43.473652 8447 master_runner.cc:315] Copying system tablet from 127.27.110.254:43403
I20260812 07:00:43.474798 8447 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: tablet copy: Beginning tablet copy session from remote peer at address 127.27.110.254:43403
I20260812 07:00:43.483608 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.475457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.27.110.254:43403)
0812 07:00:43.475593 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.475601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.475845 (+ 244us) server_negotiation.cc:407] Connection header received
0812 07:00:43.476013 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.476016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.476073 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.476158 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.476696 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.477315 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.478473 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.478723 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.478783 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.480334 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.480338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.480341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.480367 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.481871 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.482313 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.482317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.482319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.482404 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.482706 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.482709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.482711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.482893 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.483001 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.483327 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.483440 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:00:43.483623 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.475383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:44142)
0812 07:00:43.475743 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.475756 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.475891 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.476178 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.476181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.476442 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.476568 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.476575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.477521 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.477525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.478308 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.478319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.479007 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.479011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.479141 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.479758 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.479773 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.480194 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.481992 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.481996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.481999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.482208 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.482502 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.482505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.482507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.482610 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.482994 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.482997 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.483137 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.483261 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.483414 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
I20260812 07:00:43.484411 7516 tablet_copy_service.cc:140] P ca70b32cb9c149819b86023dfb5c2691: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 83e4b9bd53a44347828fd9b9eb3df233 ({username='oryx', principal='oryx/127.27.110.252@KRBTEST.COM'} at 127.0.0.1:44142)
I20260812 07:00:43.484529 7516 tablet_copy_service.cc:161] P ca70b32cb9c149819b86023dfb5c2691: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 83e4b9bd53a44347828fd9b9eb3df233 at {username='oryx', principal='oryx/127.27.110.252@KRBTEST.COM'} at 127.0.0.1:44142: session id = 83e4b9bd53a44347828fd9b9eb3df233-00000000000000000000000000000000
I20260812 07:00:43.485374 7516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 07:00:43.486517 8447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 07:00:43.488911 8447 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: tablet copy: Starting download of 0 data blocks...
I20260812 07:00:43.489122 8447 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: tablet copy: Starting download of 1 WAL segments...
I20260812 07:00:43.490603 8447 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 07:00:43.492195 7516 tablet_copy_service.cc:342] P ca70b32cb9c149819b86023dfb5c2691: Request end of tablet copy session 83e4b9bd53a44347828fd9b9eb3df233-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.110.252@KRBTEST.COM'} at 127.0.0.1:44142
I20260812 07:00:43.492300 7516 tablet_copy_service.cc:433] P ca70b32cb9c149819b86023dfb5c2691: ending tablet copy session 83e4b9bd53a44347828fd9b9eb3df233-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 83e4b9bd53a44347828fd9b9eb3df233
I20260812 07:00:43.499958 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:43.501631 8756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.501884 8757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.502135 8447 server_base.cc:1034] running on GCE node
W20260812 07:00:43.503224 8761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.503491 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:43.503815 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:43.504966 8447 hybrid_clock.cc:648] HybridClock initialized: now 1786518043504972 us; error 43 us; skew 500 ppm
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:43.506464 8447 init.cc:377] Logged in from keytab as oryx/127.27.110.252@KRBTEST.COM (short username oryx)
I20260812 07:00:43.506950 8447 webserver.cc:460] Webserver started at http://127.27.110.252:35129/ using document root <none> and password file <none>
I20260812 07:00:43.507194 8447 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.507246 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.508368 8447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:43.509425 8771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.509663 8447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:43.509760 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
uuid: "83e4b9bd53a44347828fd9b9eb3df233"
format_stamp: "Formatted at 2026-08-12 07:00:43 on dist-test-slave-w4v5"
server_key: "a880702f05a9a5a99c08c6447d82b932"
server_key_iv: "bf535b1ccfbe310cfb12b1d4b9d02ca0"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.509866 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.517935 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:43.520467 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:43.525375 8447 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.252:42849
I20260812 07:00:43.526206 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
I20260812 07:00:43.527351 8856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.252:42849 every 8 connection(s)
I20260812 07:00:43.529122 8857 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:00:43.529979 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Bootstrap starting.
I20260812 07:00:43.539317 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:00:43.539649 8857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Bootstrap complete.
I20260812 07:00:43.540311 8857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } }
I20260812 07:00:43.540469 8857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 83e4b9bd53a44347828fd9b9eb3df233, State: Initialized, Role: LEARNER
I20260812 07:00:43.540587 8857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } }
I20260812 07:00:43.541054 8857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:43.541550 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:43.541811 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: This master's current role is: LEARNER
I20260812 07:00:43.544323 8877 catalog_manager.cc:1259] Loaded cluster ID: 8fe76e40e29b4ff0950e10be7a8424de
I20260812 07:00:43.544422 8877 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: loading cluster ID for follower catalog manager: success
I20260812 07:00:43.546339 8016 heartbeater.cc:503] Master 127.27.110.254:43403 was elected leader, sending a full tablet report...
I20260812 07:00:43.546581 8877 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: acquiring CA information for follower catalog manager: success
I20260812 07:00:43.548008 8877 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 07:00:43.583240 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:43.583539 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:43.583623 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:43.587742 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:43.587823 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:43.587915 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:43.593407 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:45225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=41407
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:43.594574 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:43.595680 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:00:43.605679 8738 server_base.cc:1034] running on GCE node
W20260812 07:00:43.605582 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.605623 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.605597 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.606352 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:43.607053 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:43.608278 8738 hybrid_clock.cc:648] HybridClock initialized: now 1786518043608265 us; error 25 us; skew 500 ppm
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:43.611363 8738 init.cc:377] Logged in from keytab as oryx/127.27.110.193@KRBTEST.COM (short username oryx)
I20260812 07:00:43.612309 8738 webserver.cc:460] Webserver started at http://127.27.110.193:41407/ using document root <none> and password file <none>
I20260812 07:00:43.612887 8738 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.613004 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.616858 8738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:00:43.618983 8899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.620317 8738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:43.620468 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "676664633f2445849855b82d215f7c7f"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "a2c7e9917e78f8eb60c5d8bc51204727"
server_key_iv: "af5dd0ccee9b32d0a4e29d85b8c53de9"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.621054 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.631469 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:43.634830 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:43.635040 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:43.635727 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:43.636586 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:43.636662 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.636746 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:43.636799 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.647595 8738 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:45225
I20260812 07:00:43.647625 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:45225 every 8 connection(s)
I20260812 07:00:43.648591 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:43.653250 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 8738
I20260812 07:00:43.653510 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 7808
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.252@KRBTEST.COM
I20260812 07:00:43.667469 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.650404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:45875 (local address 127.27.110.252:42849)
0812 07:00:43.650677 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.650681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.651430 (+ 749us) server_negotiation.cc:407] Connection header received
0812 07:00:43.652542 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.652549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.652621 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.652720 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.654395 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.655394 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.656800 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.657163 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.657256 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.660970 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.660974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.660977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.661006 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.663936 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.664616 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.664621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.664626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.664710 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.665065 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.665071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.665074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.665305 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.665407 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.667062 (+ 1655us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.667224 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":113,"threads_started":1}
I20260812 07:00:43.668181 9075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.650793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.27.110.193:45875)
0812 07:00:43.651287 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.651329 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.652289 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.653039 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.653046 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.653527 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.654226 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.654246 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.655537 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.655543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.656658 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.656672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.657422 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.657428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.657629 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.658752 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.658773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.660774 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.664106 (+ 3332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.664117 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.664140 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.664494 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.664798 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.664803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.664807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.664971 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.665435 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.665443 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.665915 (+ 472us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.666213 (+ 298us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.666678 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":3,"thread_start_us":123,"threads_started":1}
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
I20260812 07:00:43.676931 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.651924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:35963 (local address 127.27.110.254:43403)
0812 07:00:43.652082 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.652087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.652661 (+ 574us) server_negotiation.cc:407] Connection header received
0812 07:00:43.652867 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.652872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.652924 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.652980 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.661841 (+ 8861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.662859 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.664003 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.664401 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.664514 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.671363 (+ 6849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.671370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.671376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.671418 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.673856 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.675069 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.675100 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.675104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.675198 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.675520 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.675525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.675527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.675768 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.675858 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.676550 (+ 692us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.676706 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:00:43.677309 9068 heartbeater.cc:347] Connected to a master server at 127.27.110.252:42849
I20260812 07:00:43.677552 9068 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:43.678133 9068 heartbeater.cc:511] Master 127.27.110.252:42849 requested a full tablet report, sending...
I20260812 07:00:43.678637 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.652285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:35963)
0812 07:00:43.652551 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.652565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.652746 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.653145 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.653150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.653544 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.661626 (+ 8082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.661642 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.663010 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.663014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.663872 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.663885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.664757 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.664766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.664921 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.669015 (+ 4094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.669036 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.671157 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.674023 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.674029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.674034 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.674493 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.675317 (+ 824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.675320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.675321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.675408 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.676241 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.676246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.676343 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.677386 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.677570 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":96,"threads_started":1}
I20260812 07:00:43.680047 9070 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43403
I20260812 07:00:43.680141 9070 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:43.680318 9070 heartbeater.cc:511] Master 127.27.110.254:43403 requested a full tablet report, sending...
I20260812 07:00:43.681280 8786 ts_manager.cc:194] Registered new tserver with Master: 676664633f2445849855b82d215f7c7f (127.27.110.193:45225)
I20260812 07:00:43.681285 7483 ts_manager.cc:194] Re-registered known tserver with Master: 676664633f2445849855b82d215f7c7f (127.27.110.193:45225)
I20260812 07:00:43.682173 7483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:35963
I20260812 07:00:43.682579 9082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.652679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.27.110.193:60313)
0812 07:00:43.658066 (+ 5387us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.658081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.658213 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.658646 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.658650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.658851 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.658984 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.658992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.660154 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.660160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.660999 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.661021 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.661728 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.661734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.661845 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.673937 (+ 12092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.673949 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.676129 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.679643 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.679648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.679650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.679914 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.680271 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.680277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.680282 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.680380 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.681739 (+ 1359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.681745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.681842 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.682254 (+ 412us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.682406 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5301,"thread_start_us":88,"threads_started":1}
I20260812 07:00:43.682619 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.652795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:60313 (local address 127.27.110.253:37331)
0812 07:00:43.652940 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.652945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.658148 (+ 5203us) server_negotiation.cc:407] Connection header received
0812 07:00:43.658315 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.658322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.658383 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.658439 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.659172 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.659962 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.661140 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.661439 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.661548 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.676837 (+ 15289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.676841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.676844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.676876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.679520 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.680028 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.680034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.680039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.680153 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.680484 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.680490 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.680495 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.680752 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.680840 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.682332 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.682451 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:00:43.683317 9071 heartbeater.cc:347] Connected to a master server at 127.27.110.253:37331
I20260812 07:00:43.683408 9071 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:43.683746 9071 heartbeater.cc:511] Master 127.27.110.253:37331 requested a full tablet report, sending...
I20260812 07:00:43.684792 8498 ts_manager.cc:194] Registered new tserver with Master: 676664633f2445849855b82d215f7c7f (127.27.110.193:45225)
I20260812 07:00:43.684823 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:44893
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:43403
--webserver_port=43903
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:00:43.776553 8258 heartbeater.cc:503] Master 127.27.110.254:43403 was elected leader, sending a full tablet report...
W20260812 07:00:43.801394 9083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:43.801640 9083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:43.801680 9083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:43.805655 9083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:43.805711 9083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:43.805794 9083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:43.811224 9083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:44893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=43903
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:43.812255 9083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:43.813311 9083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:43.823254 9093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.823267 9091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:43.823271 9095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:43.823318 9083 server_base.cc:1034] running on GCE node
I20260812 07:00:43.823896 9083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:43.824641 9083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:43.825902 9083 hybrid_clock.cc:648] HybridClock initialized: now 1786518043825861 us; error 52 us; skew 500 ppm
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:43.828985 9083 init.cc:377] Logged in from keytab as oryx/127.27.110.194@KRBTEST.COM (short username oryx)
I20260812 07:00:43.829896 9083 webserver.cc:460] Webserver started at http://127.27.110.194:43903/ using document root <none> and password file <none>
I20260812 07:00:43.830488 9083 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:43.830612 9083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:43.834517 9083 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:43.836658 9108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.837908 9083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260812 07:00:43.838042 9083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "9489275ce6a3415b856fce223d4d97c3"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "b1055c9ef4ad4a97a2b52bf679f97367"
server_key_iv: "4fb514dc73819db7de18d9333f8ea058"
server_key_version: "encryptionkey@0"
I20260812 07:00:43.838585 9083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:43.859683 9083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:43.863044 9083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:43.863308 9083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:43.863925 9083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:43.864794 9083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:43.864863 9083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.864938 9083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:43.864980 9083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:43.874092 9083 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:44893
I20260812 07:00:43.874115 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:44893 every 8 connection(s)
I20260812 07:00:43.875154 9083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:43.882277 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 9083
I20260812 07:00:43.882472 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 8028
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.252@KRBTEST.COM
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
I20260812 07:00:43.893729 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.877040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:46709 (local address 127.27.110.252:42849)
0812 07:00:43.877206 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.877213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.878595 (+ 1382us) server_negotiation.cc:407] Connection header received
0812 07:00:43.880075 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.880078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.880167 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.880247 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.881540 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.882383 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.884514 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.885034 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.885168 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.889225 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.889230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.889232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.889262 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.890860 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.891672 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.891678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.891682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.891782 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.892293 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.892299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.892301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.892557 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.892673 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.893403 (+ 730us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.893539 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:00:43.894960 9322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.877521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.27.110.194:46709)
0812 07:00:43.878399 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.878460 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.879575 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.880251 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.880254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.880805 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.881380 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.881394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.882531 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.882534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.884321 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.884337 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.885395 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.885403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.885696 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.886522 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.886550 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.889014 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.890986 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.890995 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.891016 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.891540 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.891899 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.891905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.891910 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.892152 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.892672 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.892681 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.893054 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.893343 (+ 289us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.893679 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":96,"spinlock_wait_cycles":6272,"thread_start_us":145,"threads_started":1}
I20260812 07:00:43.894987 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.877039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:55555 (local address 127.27.110.253:37331)
0812 07:00:43.877187 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.877197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.878595 (+ 1398us) server_negotiation.cc:407] Connection header received
0812 07:00:43.879828 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.879832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.879883 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.879954 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.881566 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.882192 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.887925 (+ 5733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.888161 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.888283 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.890853 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.890856 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.890859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.890890 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.892500 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.893092 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.893099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.893103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.893220 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.893651 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.893657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.893662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.893981 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.894073 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.894567 (+ 494us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.894731 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:00:43.895795 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.877521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.27.110.194:55555)
0812 07:00:43.878399 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.878460 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.879575 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.880159 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.880171 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.880824 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.881395 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.881404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.882391 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.882395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.887791 (+ 5396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.887807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.888289 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.888292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.888402 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.888909 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.888930 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.890680 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.892616 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.892622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.892627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.892949 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.893369 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.893375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.893379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.893510 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.894165 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.894171 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.894304 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.895266 (+ 962us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.895469 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":174,"spinlock_wait_cycles":84480,"thread_start_us":157,"threads_started":1}
I20260812 07:00:43.896859 9315 heartbeater.cc:347] Connected to a master server at 127.27.110.253:37331
I20260812 07:00:43.897123 9315 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:43.897640 9315 heartbeater.cc:511] Master 127.27.110.253:37331 requested a full tablet report, sending...
I20260812 07:00:43.898778 8498 ts_manager.cc:194] Registered new tserver with Master: 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893)
I20260812 07:00:43.914606 9313 heartbeater.cc:347] Connected to a master server at 127.27.110.252:42849
I20260812 07:00:43.914754 9313 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:43.915032 9313 heartbeater.cc:511] Master 127.27.110.252:42849 requested a full tablet report, sending...
I20260812 07:00:43.915971 8786 ts_manager.cc:194] Registered new tserver with Master: 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893)
Aug 12 07:00:43 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518043, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:43.920629 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:41819
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:43403
--webserver_port=46317
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:00:43.921028 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.878755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:54941 (local address 127.27.110.254:43403)
0812 07:00:43.878879 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.878883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.880136 (+ 1253us) server_negotiation.cc:407] Connection header received
0812 07:00:43.880200 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.880205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.880258 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.880331 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:43.881567 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.882385 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.914911 (+ 32526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.915251 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.915382 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.917641 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:43.917645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:43.917647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:43.917674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:43.919172 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.919599 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.919603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.919605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.919688 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:43.920026 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:43.920032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:43.920037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:43.920279 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:43.920362 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.920679 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.920856 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:43.921083 9324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.878622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.194:54941)
0812 07:00:43.880022 (+ 1400us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.880036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.880130 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.880440 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.880443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.880828 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:43.881393 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.881404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.913830 (+ 32426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.913838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.914756 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.914772 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.915427 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.915433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.915561 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.916164 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:43.916179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:43.917479 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:43.919282 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.919287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.919289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.919491 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.919782 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:43.919786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:43.919787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:43.919891 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:43.920384 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:43.920387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:43.920496 (+ 109us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.920627 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.920792 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"mutex_wait_us":56,"spinlock_wait_cycles":3200,"thread_start_us":88,"threads_started":1}
I20260812 07:00:43.921962 9314 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43403
I20260812 07:00:43.922096 9314 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:43.922310 9314 heartbeater.cc:511] Master 127.27.110.254:43403 requested a full tablet report, sending...
I20260812 07:00:43.922981 7483 ts_manager.cc:194] Re-registered known tserver with Master: 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893)
I20260812 07:00:43.923704 7483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:54941
I20260812 07:00:43.999019 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.995469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.27.110.252:42849)
0812 07:00:43.995634 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:43.995640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:43.995752 (+ 112us) server_negotiation.cc:407] Connection header received
0812 07:00:43.995874 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:43.995879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:43.995951 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:43.996069 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:43.996373 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.996924 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.997984 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.998440 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.998606 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.998780 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:43.998819 (+ 39us) server_negotiation.cc:299] Negotiation successful
0812 07:00:43.998887 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:00:43.999020 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:43.995322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:50624)
0812 07:00:43.995654 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:43.995667 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:43.995784 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:43.996090 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:43.996093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:43.996109 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:43.996224 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.996229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.997059 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.997063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.997843 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:43.997853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:43.998588 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:43.998592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:43.998683 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:43.998709 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:00:43.998777 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 07:00:43.998869 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":70,"threads_started":1}
I20260812 07:00:43.999701 8815 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260812 07:00:44.000080 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca70b32cb9c149819b86023dfb5c2691. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:44.000250 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: This master's current role is: LEARNER
W20260812 07:00:44.038305 9326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:44.038566 9326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:44.038640 9326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:44.043300 9326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:44.043362 9326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:44.043463 9326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:44.048976 9326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:41819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=46317
--tserver_master_addrs=127.27.110.254:43403,127.27.110.253:37331,127.27.110.252:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:44.050104 9326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:44.051096 9326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:44.061074 9337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:44.061065 9340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:44.061065 9338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:44.061450 9326 server_base.cc:1034] running on GCE node
I20260812 07:00:44.061821 9326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:44.062451 9326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:44.063655 9326 hybrid_clock.cc:648] HybridClock initialized: now 1786518044063636 us; error 31 us; skew 500 ppm
Aug 12 07:00:44 dist-test-slave-w4v5 krb5kdc[7411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518044, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:44.066454 9326 init.cc:377] Logged in from keytab as oryx/127.27.110.195@KRBTEST.COM (short username oryx)
I20260812 07:00:44.067416 9326 webserver.cc:460] Webserver started at http://127.27.110.195:46317/ using document root <none> and password file <none>
I20260812 07:00:44.068009 9326 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:44.068130 9326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:44.071930 9326 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260812 07:00:44.073930 9350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:44.075217 9326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:44.075345 9326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "2c6246dc472d4d4b9f04d5b72d726c90"
format_stamp: "Formatted at 2026-08-12 07:00:42 on dist-test-slave-w4v5"
server_key: "6c677c6e070633c17f0857c0b92c069b"
server_key_iv: "abf403f24c1fbdc19afbf2d8796b2f05"
server_key_version: "encryptionkey@0"
I20260812 07:00:44.075886 9326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:44.103020 9326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:44.106132 9326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:44.106348 9326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:44.106983 9326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:44.107892 9326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:44.107982 9326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:44.108052 9326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:44.108076 9326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:44.117297 9326 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:41819
I20260812 07:00:44.117365 9573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:41819 every 8 connection(s)
I20260812 07:00:44.118497 9326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:44.128587 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 9326
Aug 12 07:00:44 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518044, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.252@KRBTEST.COM
Aug 12 07:00:44 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518044, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
Aug 12 07:00:44 dist-test-slave-w4v5 krb5kdc[7411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518044, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.253@KRBTEST.COM
I20260812 07:00:44.136253 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.120277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:57991 (local address 127.27.110.252:42849)
0812 07:00:44.120418 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.120422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.121472 (+ 1050us) server_negotiation.cc:407] Connection header received
0812 07:00:44.122484 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.122487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.122548 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.122655 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.124132 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.124750 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.126588 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.126934 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.127016 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.131233 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.131238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.131240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.131268 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.134258 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134741 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.134745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.134747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.134835 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.135268 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.135271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.135273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.135484 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.135561 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.135900 (+ 339us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.136061 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:00:44.137226 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.120679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.27.110.195:57991)
0812 07:00:44.121373 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.121387 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.122285 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.122848 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.122855 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.123500 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:44.123981 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.123994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.124899 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.124905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.126460 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.126472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.127041 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.127045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.127147 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.127835 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:44.127860 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:44.131042 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:44.134370 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:44.134375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:44.134628 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:44.134939 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134941 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:44.134943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:44.135135 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:44.135639 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:44.135641 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:44.135729 (+ 88us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.136396 (+ 667us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.136539 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":182,"spinlock_wait_cycles":4352,"thread_start_us":140,"threads_started":1}
I20260812 07:00:44.137548 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.121682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.27.110.195:45515)
0812 07:00:44.121952 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.121964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.122291 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.122849 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.122856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.123462 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:44.123989 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.123998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.125063 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.125066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.126460 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.126472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.127559 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.127562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.127662 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.127976 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:44.127993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:44.131411 (+ 3418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:44.133528 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:44.133534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:44.133552 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:44.133923 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:44.134605 (+ 682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:44.134610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:44.134784 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:44.135326 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:44.135332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:44.135626 (+ 294us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.135820 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.136144 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":257,"spinlock_wait_cycles":27008,"thread_start_us":98,"threads_started":1}
I20260812 07:00:44.138409 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.121813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:45515 (local address 127.27.110.253:37331)
0812 07:00:44.121932 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.121939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.122018 (+ 79us) server_negotiation.cc:407] Connection header received
0812 07:00:44.122513 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.122517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.122575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.122630 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.124170 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.124953 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.127169 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.127407 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.127518 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.131579 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.131584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.131587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.131619 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.133391 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134409 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.134412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.134414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.134502 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134888 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.134891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.134893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.135165 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.135266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.138078 (+ 2812us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.138218 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:44.138845 9574 heartbeater.cc:347] Connected to a master server at 127.27.110.252:42849
I20260812 07:00:44.139092 9577 heartbeater.cc:347] Connected to a master server at 127.27.110.253:37331
I20260812 07:00:44.139119 9574 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:44.139189 9577 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:44.139735 9574 heartbeater.cc:511] Master 127.27.110.252:42849 requested a full tablet report, sending...
I20260812 07:00:44.139823 9577 heartbeater.cc:511] Master 127.27.110.253:37331 requested a full tablet report, sending...
I20260812 07:00:44.140655 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.120679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:37585)
0812 07:00:44.121198 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.121250 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.122290 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.122849 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.122856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.123475 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:44.123984 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.123998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.125101 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.125107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.126256 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.126266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.126866 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.126871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.127104 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.128242 (+ 1138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:44.128256 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:44.130503 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:44.134038 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:44.134041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:44.134043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:44.134297 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:44.139453 (+ 5156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:44.139456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:44.139458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:44.139572 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:44.140111 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:44.140114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:44.140210 (+ 96us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.140310 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.140464 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":278,"spinlock_wait_cycles":15232,"thread_start_us":127,"threads_started":1}
I20260812 07:00:44.141079 8498 ts_manager.cc:194] Registered new tserver with Master: 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819)
I20260812 07:00:44.141295 7541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.120346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:37585 (local address 127.27.110.254:43403)
0812 07:00:44.120481 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.120485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.121359 (+ 874us) server_negotiation.cc:407] Connection header received
0812 07:00:44.122519 (+ 1160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.122522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.122575 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.122630 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.124153 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.124958 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.126385 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.126693 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.126797 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.130760 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.130764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.130767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.130797 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.133414 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.139237 (+ 5823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.139240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.139242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.139338 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.139716 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.139719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.139723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.139943 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.140040 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.141018 (+ 978us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.141138 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:00:44.141634 8786 ts_manager.cc:194] Registered new tserver with Master: 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819)
I20260812 07:00:44.141987 9575 heartbeater.cc:347] Connected to a master server at 127.27.110.254:43403
I20260812 07:00:44.142076 9575 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:44.142270 9575 heartbeater.cc:511] Master 127.27.110.254:43403 requested a full tablet report, sending...
I20260812 07:00:44.142882 7483 ts_manager.cc:194] Re-registered known tserver with Master: 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819)
I20260812 07:00:44.143666 7483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:37585
I20260812 07:00:44.146569 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.131156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.27.110.254:43403)
0812 07:00:44.132820 (+ 1664us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.132825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.133066 (+ 241us) server_negotiation.cc:407] Connection header received
0812 07:00:44.133109 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.133112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.133170 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.133256 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.134229 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.135028 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.136419 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.136768 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.136843 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.138821 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.138827 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.138832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.138861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.140664 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.141131 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.141135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.141137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.141245 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.142648 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.142652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.142654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.144116 (+ 1462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.145512 (+ 1396us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.146113 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.146248 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1571,"thread_start_us":78,"threads_started":1}
I20260812 07:00:44.146711 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.130899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.27.110.252:42849)
0812 07:00:44.132966 (+ 2067us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.132971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.132984 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:00:44.133022 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.133025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.133072 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.134097 (+ 1025us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.134204 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.134960 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.136212 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.136616 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.137082 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.142903 (+ 5821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.142909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.142912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.142941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.144731 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.145215 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.145219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.145221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.145308 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.145628 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.145631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.145633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.145860 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.145991 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.146327 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.146458 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1994,"thread_start_us":103,"threads_started":1}
I20260812 07:00:44.150257 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.132509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.27.110.253:37331)
0812 07:00:44.138592 (+ 6083us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.138596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.141644 (+ 3048us) server_negotiation.cc:407] Connection header received
0812 07:00:44.141876 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.141880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.141921 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.141967 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.142954 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.143797 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.145059 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.145318 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.145432 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.147094 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.147098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.147101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.147125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.148724 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.149210 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.149213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.149214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.149282 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.149553 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.149555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.149556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.149693 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.149781 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.150054 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.150142 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6008,"thread_start_us":95,"threads_started":1}
I20260812 07:00:44.149582 7483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:44.153141 7483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:44.155722 9328 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: NON_VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: true } } }
I20260812 07:00:44.155855 9328 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:44.159161 9328 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: attempting to promote NON_VOTER 83e4b9bd53a44347828fd9b9eb3df233 to VOTER
I20260812 07:00:44.159461 9328 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } }
I20260812 07:00:44.160566 8815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 LEARNER]: Refusing update from remote peer ca70b32cb9c149819b86023dfb5c2691: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 07:00:44.160724 8515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Refusing update from remote peer ca70b32cb9c149819b86023dfb5c2691: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260812 07:00:44.161124 9615 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 07:00:44.161324 9611 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260812 07:00:44.163394 9328 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.165262 9630 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.165371 9630 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:44.165482 9630 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.165546 9630 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:44.165591 9328 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.165714 9328 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:44.166419 8515 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.167609 8815 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.167986 9632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.168155 9632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:44.169559 8861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ca70b32cb9c149819b86023dfb5c2691" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:44.169680 8861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:44.181705 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.173743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.27.110.193:45225)
0812 07:00:44.174067 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.174074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.176571 (+ 2497us) server_negotiation.cc:407] Connection header received
0812 07:00:44.176671 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.176677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.176858 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.176986 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:44.177325 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.178517 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.180068 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.180940 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.181052 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.181212 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.181505 (+ 293us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.181594 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":106,"threads_started":1}
I20260812 07:00:44.181699 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.173411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:45225 (local address 127.0.0.1:38304)
0812 07:00:44.176375 (+ 2964us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.176389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.176478 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.177116 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.177119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.177136 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:44.177212 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.177216 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.178652 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.178655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.179947 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.179956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.181274 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.181277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.181363 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.181387 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.181464 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.181536 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2906,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260812 07:00:44.182575 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.173235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:44893 (local address 127.0.0.1:41906)
0812 07:00:44.175251 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.175263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.175375 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.176062 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.176067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.176090 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:44.176204 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.176210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.177548 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.177551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.181028 (+ 3477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.181039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.182164 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.182178 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.182277 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.182303 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.182376 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.182445 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1959,"mutex_wait_us":23,"thread_start_us":94,"threads_started":1}
I20260812 07:00:44.182761 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.172793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:41819 (local address 127.0.0.1:33174)
0812 07:00:44.172907 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.172925 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.173005 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.176128 (+ 3123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.176132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.176152 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:44.176248 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.176253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.178502 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.178505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.179815 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.179826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.182361 (+ 2535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.182365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.182450 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.182470 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.182536 (+ 66us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.182624 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":57}
I20260812 07:00:44.182787 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.173197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33174 (local address 127.27.110.195:41819)
0812 07:00:44.175680 (+ 2483us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.175693 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.175715 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:00:44.175790 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.175800 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.175981 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.176137 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:44.176389 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.178373 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.179962 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.180876 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.182380 (+ 1504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.182507 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.182564 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.182655 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2340,"thread_start_us":114,"threads_started":1}
I20260812 07:00:44.183058 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.173396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.27.110.194:44893)
0812 07:00:44.173745 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.173756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.175362 (+ 1606us) server_negotiation.cc:407] Connection header received
0812 07:00:44.175497 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.175504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.175741 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.175901 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:44.176342 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.177427 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.181160 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.181995 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.182130 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.182248 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.182796 (+ 548us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.182902 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":123,"threads_started":1}
I20260812 07:00:44.185429 9455 tablet_service.cc:1467] Processing CreateTablet for tablet fb3603d13b874627b0fb24962c487bc3 (DEFAULT_TABLE table=test-table [id=37103a4dc5c64c08b0e500f07dce208a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:44.185410 8972 tablet_service.cc:1467] Processing CreateTablet for tablet fb3603d13b874627b0fb24962c487bc3 (DEFAULT_TABLE table=test-table [id=37103a4dc5c64c08b0e500f07dce208a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:44.185536 9184 tablet_service.cc:1467] Processing CreateTablet for tablet fb3603d13b874627b0fb24962c487bc3 (DEFAULT_TABLE table=test-table [id=37103a4dc5c64c08b0e500f07dce208a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:44.186460 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3603d13b874627b0fb24962c487bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:44.186466 9184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3603d13b874627b0fb24962c487bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:44.186466 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb3603d13b874627b0fb24962c487bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:44.193951 9642 tablet_bootstrap.cc:492] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Bootstrap starting.
I20260812 07:00:44.194980 9645 tablet_bootstrap.cc:492] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Bootstrap starting.
I20260812 07:00:44.195413 9644 tablet_bootstrap.cc:492] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Bootstrap starting.
I20260812 07:00:44.196116 9642 tablet_bootstrap.cc:654] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:44.196843 9642 log.cc:824] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:44.197104 9645 tablet_bootstrap.cc:654] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:44.197382 9644 tablet_bootstrap.cc:654] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:44.197983 9645 log.cc:824] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:44.198110 9644 log.cc:824] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:44.198952 9642 tablet_bootstrap.cc:492] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: No bootstrap required, opened a new log
I20260812 07:00:44.199164 9642 ts_tablet_manager.cc:1397] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:44.199880 9644 tablet_bootstrap.cc:492] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: No bootstrap required, opened a new log
I20260812 07:00:44.200009 9645 tablet_bootstrap.cc:492] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: No bootstrap required, opened a new log
I20260812 07:00:44.200063 9644 ts_tablet_manager.cc:1397] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 07:00:44.200213 9645 ts_tablet_manager.cc:1397] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 07:00:44.202968 9644 raft_consensus.cc:348] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.203277 9644 raft_consensus.cc:374] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:44.203109 9642 raft_consensus.cc:348] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.203346 9644 raft_consensus.cc:729] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676664633f2445849855b82d215f7c7f, State: Initialized, Role: FOLLOWER
I20260812 07:00:44.203356 9642 raft_consensus.cc:374] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:44.203405 9642 raft_consensus.cc:729] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6246dc472d4d4b9f04d5b72d726c90, State: Initialized, Role: FOLLOWER
I20260812 07:00:44.203821 9644 consensus_queue.cc:260] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.203915 9642 consensus_queue.cc:260] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.204787 9644 ts_tablet_manager.cc:1428] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:44.204639 9645 raft_consensus.cc:348] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.204841 9642 ts_tablet_manager.cc:1428] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:00:44.204875 9645 raft_consensus.cc:374] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:44.204921 9645 raft_consensus.cc:729] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9489275ce6a3415b856fce223d4d97c3, State: Initialized, Role: FOLLOWER
I20260812 07:00:44.205010 9070 heartbeater.cc:503] Master 127.27.110.254:43403 was elected leader, sending a full tablet report...
I20260812 07:00:44.205998 9575 heartbeater.cc:503] Master 127.27.110.254:43403 was elected leader, sending a full tablet report...
I20260812 07:00:44.207557 9645 consensus_queue.cc:260] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.208528 9645 ts_tablet_manager.cc:1428] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260812 07:00:44.208869 9314 heartbeater.cc:503] Master 127.27.110.254:43403 was elected leader, sending a full tablet report...
W20260812 07:00:44.370164 9579 tablet.cc:2369] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:00:44.377182 9319 tablet.cc:2369] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:44.379772 9650 raft_consensus.cc:482] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:44.379997 9650 raft_consensus.cc:504] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.381285 9650 leader_election.cc:288] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819), 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893)
I20260812 07:00:44.385278 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.381586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:47011 (local address 127.27.110.195:41819)
0812 07:00:44.381734 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.381739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.381993 (+ 254us) server_negotiation.cc:407] Connection header received
0812 07:00:44.382172 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.382175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.382227 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.382313 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:44.382592 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.383179 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.384305 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.384707 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.384832 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.384886 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.385066 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.385146 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:00:44.385295 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.381656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:37261 (local address 127.27.110.194:44893)
0812 07:00:44.381795 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.381801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.382033 (+ 232us) server_negotiation.cc:407] Connection header received
0812 07:00:44.382217 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.382220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.382262 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.382350 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:44.382656 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.383209 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.384319 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.384723 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.384851 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.384905 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.385096 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.385167 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:00:44.385330 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.381542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:44893 (local address 127.27.110.193:37261)
0812 07:00:44.381919 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.381930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.382077 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.382362 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.382387 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.382404 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:44.382527 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.382532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.383317 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.383323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.384189 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.384198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.384847 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.384851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.384961 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.384984 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.385077 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.385166 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":42,"spinlock_wait_cycles":9472,"thread_start_us":136,"threads_started":1}
I20260812 07:00:44.385306 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.381526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:41819 (local address 127.27.110.193:47011)
0812 07:00:44.381896 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:44.381908 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:44.382055 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:44.382327 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:44.382330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:44.382346 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:44.382476 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.382481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.383303 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.383307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.384179 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:44.384189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.384827 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.384831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:44.384930 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.384960 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:00:44.385048 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 07:00:44.385125 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":5,"thread_start_us":147,"threads_started":1}
I20260812 07:00:44.385959 9220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3603d13b874627b0fb24962c487bc3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9489275ce6a3415b856fce223d4d97c3" is_pre_election: true
I20260812 07:00:44.386153 9491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3603d13b874627b0fb24962c487bc3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" is_pre_election: true
I20260812 07:00:44.386256 9220 raft_consensus.cc:2454] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 0.
I20260812 07:00:44.386456 9491 raft_consensus.cc:2454] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 0.
I20260812 07:00:44.386775 8901 leader_election.cc:302] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 676664633f2445849855b82d215f7c7f, 9489275ce6a3415b856fce223d4d97c3; no voters:
I20260812 07:00:44.387125 9650 raft_consensus.cc:2781] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:44.387215 9650 raft_consensus.cc:482] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:44.387272 9650 raft_consensus.cc:3037] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:44.388348 9650 raft_consensus.cc:504] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.388743 9650 leader_election.cc:288] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 election: Requested vote from peers 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819), 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893)
I20260812 07:00:44.389055 9491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3603d13b874627b0fb24962c487bc3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6246dc472d4d4b9f04d5b72d726c90"
I20260812 07:00:44.389127 9220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb3603d13b874627b0fb24962c487bc3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9489275ce6a3415b856fce223d4d97c3"
I20260812 07:00:44.389180 9491 raft_consensus.cc:3037] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:44.389269 9220 raft_consensus.cc:3037] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:44.390249 9491 raft_consensus.cc:2454] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 1.
I20260812 07:00:44.390409 9220 raft_consensus.cc:2454] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 1.
I20260812 07:00:44.390570 8904 leader_election.cc:302] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6246dc472d4d4b9f04d5b72d726c90, 676664633f2445849855b82d215f7c7f; no voters:
I20260812 07:00:44.390745 9650 raft_consensus.cc:2781] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:44.390975 9650 raft_consensus.cc:686] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 1 LEADER]: Becoming Leader. State: Replica: 676664633f2445849855b82d215f7c7f, State: Running, Role: LEADER
I20260812 07:00:44.391346 9650 consensus_queue.cc:237] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } }
I20260812 07:00:44.395125 7482 catalog_manager.cc:5495] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 676664633f2445849855b82d215f7c7f (127.27.110.193). New cstate: current_term: 1 leader_uuid: "676664633f2445849855b82d215f7c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } health_report { overall_health: UNKNOWN } } }
W20260812 07:00:44.400187 9073 tablet.cc:2369] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:44.412343 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.408920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.27.110.193:45225)
0812 07:00:44.409056 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.409059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.409073 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:44.409183 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.409185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.409234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.409380 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:44.409616 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.410120 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.411025 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.411321 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.411403 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.411664 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:44.412049 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:44.412139 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.412196 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.412243 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:00:44.418113 9491 raft_consensus.cc:1261] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Refusing update from remote peer 676664633f2445849855b82d215f7c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:44.418100 9220 raft_consensus.cc:1261] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Refusing update from remote peer 676664633f2445849855b82d215f7c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:44.418879 9658 consensus_queue.cc:1035] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:44.419068 9650 consensus_queue.cc:1035] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:44.426585 9661 mvcc.cc:205] Tried to move back new op lower bound from 7317577909929332736 to 7317577909829365760. Current Snapshot: MvccSnapshot[applied={T|T < 7317577909929332736}]
I20260812 07:00:44.428234 9663 mvcc.cc:205] Tried to move back new op lower bound from 7317577909929332736 to 7317577909829365760. Current Snapshot: MvccSnapshot[applied={T|T < 7317577909929332736}]
I20260812 07:00:44.429210 9664 mvcc.cc:205] Tried to move back new op lower bound from 7317577909929332736 to 7317577909829365760. Current Snapshot: MvccSnapshot[applied={T|T < 7317577909929332736}]
I20260812 07:00:44.437031 7482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:44.437177 7482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:44.441340 7482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca70b32cb9c149819b86023dfb5c2691: Sending DeleteTablet for 3 replicas of tablet fb3603d13b874627b0fb24962c487bc3
I20260812 07:00:44.442297 9184 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3603d13b874627b0fb24962c487bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:44 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:41906
I20260812 07:00:44.442301 8972 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3603d13b874627b0fb24962c487bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:44 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:38304
I20260812 07:00:44.442773 9673 tablet_replica.cc:330] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: stopping tablet replica
I20260812 07:00:44.443288 9673 raft_consensus.cc:2229] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:44.443384 9455 tablet_service.cc:1514] Processing DeleteTablet for tablet fb3603d13b874627b0fb24962c487bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:44 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:33174
I20260812 07:00:44.443938 9673 raft_consensus.cc:2258] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:44.445605 9677 tablet_replica.cc:330] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: stopping tablet replica
I20260812 07:00:44.445735 9673 ts_tablet_manager.cc:1905] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:44.445909 9677 raft_consensus.cc:2229] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:44.445943 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 7431
I20260812 07:00:44.454017 9677 raft_consensus.cc:2258] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:44.456068 9674 tablet_replica.cc:330] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: stopping tablet replica
I20260812 07:00:44.456472 9674 raft_consensus.cc:2229] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:44.456902 9674 raft_consensus.cc:2258] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:44.457819 9677 ts_tablet_manager.cc:1905] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:44.459746 9673 ts_tablet_manager.cc:1918] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:44.459918 9673 log.cc:1196] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/fb3603d13b874627b0fb24962c487bc3
I20260812 07:00:44.460358 9673 ts_tablet_manager.cc:1939] T fb3603d13b874627b0fb24962c487bc3 P 9489275ce6a3415b856fce223d4d97c3: Deleting consensus metadata
I20260812 07:00:44.461385 9674 ts_tablet_manager.cc:1905] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:44.462934 9677 ts_tablet_manager.cc:1918] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260812 07:00:44.463255 9116 connection.cc:441] server connection from 127.0.0.1:41906 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41906 (request call id 1) could send its response
I20260812 07:00:44.463590 9677 log.cc:1196] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/fb3603d13b874627b0fb24962c487bc3
I20260812 07:00:44.464059 9677 ts_tablet_manager.cc:1939] T fb3603d13b874627b0fb24962c487bc3 P 2c6246dc472d4d4b9f04d5b72d726c90: Deleting consensus metadata
W20260812 07:00:44.465641 9351 connection.cc:441] server connection from 127.0.0.1:33174 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33174 (request call id 1) could send its response
I20260812 07:00:44.466053 9674 ts_tablet_manager.cc:1918] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:44.466178 9674 log.cc:1196] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/fb3603d13b874627b0fb24962c487bc3
I20260812 07:00:44.466583 9674 ts_tablet_manager.cc:1939] T fb3603d13b874627b0fb24962c487bc3 P 676664633f2445849855b82d215f7c7f: Deleting consensus metadata
W20260812 07:00:44.467837 8902 connection.cc:441] server connection from 127.0.0.1:38304 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38304 (request call id 1) could send its response
I20260812 07:00:44.475174 8704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.443854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.27.110.253:37331)
0812 07:00:44.444108 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.444114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.453666 (+ 9552us) server_negotiation.cc:407] Connection header received
0812 07:00:44.454551 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.454557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.454623 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.454680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.457209 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.458154 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.462277 (+ 4123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.462635 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.462721 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.469832 (+ 7111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.469836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.469839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.469870 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.471733 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.472251 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.472257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.472258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.472357 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.472708 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.472711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.472713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.472936 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.473050 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.473577 (+ 527us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.474964 (+ 1387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260812 07:00:44.478349 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.466591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33028 (local address 127.27.110.253:37331)
0812 07:00:44.466742 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.466748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.467253 (+ 505us) server_negotiation.cc:407] Connection header received
0812 07:00:44.467388 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.467392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.467452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.467546 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.468987 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.469874 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.470917 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.471315 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.471414 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.473071 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.473075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.473078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.473103 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.474859 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.476960 (+ 2101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.476963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.476965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.477049 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.477341 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.477344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.477346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.477539 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.477698 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.478079 (+ 381us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.478202 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:00:44.481110 9074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.467693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.27.110.252:42849)
0812 07:00:44.467842 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.467848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.471375 (+ 3527us) server_negotiation.cc:407] Connection header received
0812 07:00:44.471581 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.471586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.471652 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.471710 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.472861 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.473608 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.474818 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.475210 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.475285 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.476828 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.476831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.476833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.476859 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.478896 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.479495 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.479503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.479506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.479589 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.479942 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.479946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.479947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.480186 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.480320 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.480711 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.480943 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:00:44.482775 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:44.444208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.27.110.252:42849)
0812 07:00:44.444529 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:44.444534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:44.473892 (+ 29358us) server_negotiation.cc:407] Connection header received
0812 07:00:44.474042 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:44.474045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:44.474090 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:44.474142 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:44.474867 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.475583 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.476808 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:44.477179 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:44.477273 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:44.479274 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:44.479279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:44.479282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:44.479311 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:44.481062 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.481530 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.481534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.481535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.481622 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:44.481976 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:44.481978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:44.481979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:44.482132 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:44.482270 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:44.482496 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 07:00:44.482622 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260812 07:00:45.431469 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.430951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:51493)
0812 07:00:45.431266 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.431359 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":80,"threads_started":1}
W20260812 07:00:45.431900 9575 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:43403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.433574 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.433077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.194:45227)
0812 07:00:45.433366 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.433467 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260812 07:00:45.434196 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.433779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:43767)
0812 07:00:45.434006 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.434099 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
W20260812 07:00:45.434182 9314 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:43403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
W20260812 07:00:45.434465 9070 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:43403 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.942055 9693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca70b32cb9c149819b86023dfb5c2691)
I20260812 07:00:45.942332 9693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } }
I20260812 07:00:45.943522 9693 leader_election.cc:288] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403), 0d95f9f8d7ea48519c09223c32d7337a (127.27.110.253:37331)
I20260812 07:00:45.944118 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.943674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50600)
0812 07:00:45.943915 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.944001 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
W20260812 07:00:45.944332 8775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.945200 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.944933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50608)
0812 07:00:45.945051 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.945123 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 07:00:45.945228 9704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ca70b32cb9c149819b86023dfb5c2691)
W20260812 07:00:45.945312 8775 leader_election.cc:334] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403): Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.945420 9704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } }
I20260812 07:00:45.946332 9704 leader_election.cc:288] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403), 83e4b9bd53a44347828fd9b9eb3df233 (127.27.110.252:42849)
I20260812 07:00:45.946832 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.946422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50620)
0812 07:00:45.946673 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.946746 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
W20260812 07:00:45.947206 8480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.948048 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.947832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50624)
0812 07:00:45.947939 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.947987 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 07:00:45.948242 8480 leader_election.cc:334] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403): Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.948853 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.944119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.27.110.253:37331)
0812 07:00:45.944799 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:45.944806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:45.944822 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:45.944876 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:45.944879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:45.944926 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:45.945014 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:45.945466 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.946235 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.947748 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.948363 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.948505 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:45.948556 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:45.948708 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 07:00:45.948767 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":590,"thread_start_us":85,"threads_started":1}
I20260812 07:00:45.948890 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.944033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:37331 (local address 127.0.0.1:53882)
0812 07:00:45.944259 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.944273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:45.944368 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:45.945204 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:45.945210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:45.945237 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:45.945353 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:45.945360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.946375 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.946379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:45.947629 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:45.947637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.948475 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.948481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:45.948580 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:45.948608 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:00:45.948699 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 07:00:45.948768 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
I20260812 07:00:45.949369 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83e4b9bd53a44347828fd9b9eb3df233" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d95f9f8d7ea48519c09223c32d7337a" is_pre_election: true
I20260812 07:00:45.949568 8515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83e4b9bd53a44347828fd9b9eb3df233 in term 1.
I20260812 07:00:45.949939 8775 leader_election.cc:302] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d95f9f8d7ea48519c09223c32d7337a, 83e4b9bd53a44347828fd9b9eb3df233; no voters: ca70b32cb9c149819b86023dfb5c2691
I20260812 07:00:45.950170 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 07:00:45.950237 9693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ca70b32cb9c149819b86023dfb5c2691)
I20260812 07:00:45.950289 9693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:00:45.951207 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.946482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:42849 (local address 127.0.0.1:60876)
0812 07:00:45.946910 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.946926 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:45.947018 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:45.947400 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:45.947403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:45.947419 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:45.947496 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:45.947501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.948461 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.948465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:45.949977 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:45.949988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.950783 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.950787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:45.950871 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:45.950895 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:00:45.950976 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 07:00:45.951041 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":19,"thread_start_us":47,"threads_started":1}
I20260812 07:00:45.951227 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.946836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.27.110.252:42849)
0812 07:00:45.947130 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:45.947139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:45.947152 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:00:45.947190 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:45.947192 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:45.947228 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:45.947307 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:45.947624 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.948361 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.950120 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:45.950656 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:45.950751 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:45.950789 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:45.950999 (+ 210us) server_negotiation.cc:299] Negotiation successful
0812 07:00:45.951057 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
I20260812 07:00:45.951668 9693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } }
I20260812 07:00:45.951746 8815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d95f9f8d7ea48519c09223c32d7337a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "83e4b9bd53a44347828fd9b9eb3df233" is_pre_election: true
I20260812 07:00:45.951987 8815 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d95f9f8d7ea48519c09223c32d7337a in current term 2: Already voted for candidate 83e4b9bd53a44347828fd9b9eb3df233 in this term.
I20260812 07:00:45.952098 9693 leader_election.cc:288] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [CANDIDATE]: Term 2 election: Requested vote from peers ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403), 0d95f9f8d7ea48519c09223c32d7337a (127.27.110.253:37331)
I20260812 07:00:45.952422 8472 leader_election.cc:302] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d95f9f8d7ea48519c09223c32d7337a; no voters: 83e4b9bd53a44347828fd9b9eb3df233, ca70b32cb9c149819b86023dfb5c2691
I20260812 07:00:45.952519 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.952173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50630)
0812 07:00:45.952396 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.952457 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260812 07:00:45.952657 9704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:00:45.953014 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "83e4b9bd53a44347828fd9b9eb3df233" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d95f9f8d7ea48519c09223c32d7337a"
I20260812 07:00:45.953126 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:45.952936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50644)
0812 07:00:45.953025 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:45.953070 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 07:00:45.953228 8775 leader_election.cc:334] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403): Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
I20260812 07:00:45.953742 9704 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:00:45.954664 8515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83e4b9bd53a44347828fd9b9eb3df233 in term 2.
I20260812 07:00:45.954984 8775 leader_election.cc:302] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d95f9f8d7ea48519c09223c32d7337a, 83e4b9bd53a44347828fd9b9eb3df233; no voters: ca70b32cb9c149819b86023dfb5c2691
I20260812 07:00:45.955255 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:00:45.955479 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [term 2 LEADER]: Becoming Leader. State: Replica: 83e4b9bd53a44347828fd9b9eb3df233, State: Running, Role: LEADER
I20260812 07:00:45.955772 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } }
I20260812 07:00:45.957301 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83e4b9bd53a44347828fd9b9eb3df233. Latest consensus state: current_term: 2 leader_uuid: "83e4b9bd53a44347828fd9b9eb3df233" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:45.957398 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:45.957666 9714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:45.960093 9714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:45.960513 9714 catalog_manager.cc:1259] Loaded cluster ID: 8fe76e40e29b4ff0950e10be7a8424de
I20260812 07:00:45.960561 9714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:45.961122 9714 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:45.961524 9714 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Loaded TSK: 0
I20260812 07:00:45.962188 9714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:00:45.999873 8800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:46.002997 8800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:46.006292 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.006024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50662)
0812 07:00:46.006162 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.006223 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 07:00:46.006397 8515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [term 2 FOLLOWER]: Refusing update from remote peer 83e4b9bd53a44347828fd9b9eb3df233: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 07:00:46.007196 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.006925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.0.0.1:50668)
0812 07:00:46.007035 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.007139 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 07:00:46.007293 9709 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 07:00:46.007499 8775 consensus_peers.cc:487] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 -> Peer ca70b32cb9c149819b86023dfb5c2691 (127.27.110.254:43403): Couldn't send request to peer ca70b32cb9c149819b86023dfb5c2691. Status: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 07:00:46.008898 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83e4b9bd53a44347828fd9b9eb3df233. Latest consensus state: current_term: 2 leader_uuid: "83e4b9bd53a44347828fd9b9eb3df233" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:46.009017 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:46.010387 9709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "83e4b9bd53a44347828fd9b9eb3df233" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:46.010556 9709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:46.010820 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "83e4b9bd53a44347828fd9b9eb3df233" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "ca70b32cb9c149819b86023dfb5c2691" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } peers { permanent_uuid: "0d95f9f8d7ea48519c09223c32d7337a" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 37331 } attrs { promote: false } } peers { permanent_uuid: "83e4b9bd53a44347828fd9b9eb3df233" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 42849 } attrs { promote: false } } }
I20260812 07:00:46.010980 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95f9f8d7ea48519c09223c32d7337a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:00:46.021698 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.016941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45470 (local address 127.27.110.194:44893)
0812 07:00:46.017906 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:46.017911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:46.017924 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:00:46.017989 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:46.017992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:46.018041 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:46.018499 (+ 458us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:46.018599 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.019173 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.020416 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.021055 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.021188 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.021244 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:46.021471 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 07:00:46.021556 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":125,"threads_started":1}
I20260812 07:00:46.021688 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.016833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:44893 (local address 127.0.0.1:45470)
0812 07:00:46.016998 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.017012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:46.017119 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:46.018160 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:46.018164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:46.018178 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:46.018278 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:46.018284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.019329 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.019335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:46.020256 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:46.020269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.021197 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.021201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:46.021296 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.021320 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:00:46.021451 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 07:00:46.021541 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":81}
I20260812 07:00:46.022063 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.016961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:41819 (local address 127.0.0.1:51416)
0812 07:00:46.017213 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.017227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:46.017313 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:46.018219 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:46.018222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:46.018245 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:46.018401 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:46.018404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.019384 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.019387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:46.020708 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:46.020719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.021663 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.021666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:46.021761 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.021784 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:00:46.021841 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 07:00:46.021920 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":200}
I20260812 07:00:46.022105 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.017230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:45225 (local address 127.0.0.1:33668)
0812 07:00:46.017497 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.017512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:46.017602 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:46.018019 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:46.018022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:46.018039 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:46.018176 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:46.018183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.019276 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.019283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:46.020708 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:46.020719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.021756 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.021759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:46.021852 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.021870 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 07:00:46.021931 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 07:00:46.021989 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":101,"thread_start_us":104,"threads_started":1}
I20260812 07:00:46.022598 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.017371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.27.110.193:45225)
0812 07:00:46.017659 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:46.017665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:46.017681 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:46.017738 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:46.017741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:46.017792 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:46.017887 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:46.018284 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.019154 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.020868 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.021522 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.021654 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.021706 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:46.022349 (+ 643us) server_negotiation.cc:299] Negotiation successful
0812 07:00:46.022453 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260812 07:00:46.022830 9184 tablet_service.cc:1467] Processing CreateTablet for tablet e952e8de33be4786b8b3a6c0fdcbd2d3 (DEFAULT_TABLE table=test-table [id=3ddc195103644e57b3f3b02d7069795e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:46.022892 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.017557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.27.110.195:41819)
0812 07:00:46.017938 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:46.017944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:46.017961 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:00:46.018018 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:46.018021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:46.018073 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:46.018290 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:46.018496 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.019294 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.020861 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.021515 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.022569 (+ 1054us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.022640 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:46.022682 (+ 42us) server_negotiation.cc:299] Negotiation successful
0812 07:00:46.022762 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":79,"threads_started":1}
I20260812 07:00:46.023106 8972 tablet_service.cc:1467] Processing CreateTablet for tablet e952e8de33be4786b8b3a6c0fdcbd2d3 (DEFAULT_TABLE table=test-table [id=3ddc195103644e57b3f3b02d7069795e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:46.023185 9184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e952e8de33be4786b8b3a6c0fdcbd2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:46.023375 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e952e8de33be4786b8b3a6c0fdcbd2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:46.025986 9728 tablet_bootstrap.cc:492] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Bootstrap starting.
I20260812 07:00:46.026149 9455 tablet_service.cc:1467] Processing CreateTablet for tablet e952e8de33be4786b8b3a6c0fdcbd2d3 (DEFAULT_TABLE table=test-table [id=3ddc195103644e57b3f3b02d7069795e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:46.026470 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e952e8de33be4786b8b3a6c0fdcbd2d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:46.026894 9728 tablet_bootstrap.cc:654] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:46.027287 9729 tablet_bootstrap.cc:492] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: Bootstrap starting.
I20260812 07:00:46.028201 9729 tablet_bootstrap.cc:654] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:46.029148 9734 tablet_bootstrap.cc:492] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Bootstrap starting.
I20260812 07:00:46.029649 9729 tablet_bootstrap.cc:492] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: No bootstrap required, opened a new log
I20260812 07:00:46.029762 9729 ts_tablet_manager.cc:1397] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:46.030196 9734 tablet_bootstrap.cc:654] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:46.030277 9729 raft_consensus.cc:348] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.030411 9729 raft_consensus.cc:374] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:46.030469 9729 raft_consensus.cc:729] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676664633f2445849855b82d215f7c7f, State: Initialized, Role: FOLLOWER
I20260812 07:00:46.030616 9729 consensus_queue.cc:260] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.030979 9729 ts_tablet_manager.cc:1428] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:46.031545 9728 tablet_bootstrap.cc:492] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: No bootstrap required, opened a new log
I20260812 07:00:46.031652 9728 ts_tablet_manager.cc:1397] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260812 07:00:46.031889 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.031324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:32877)
0812 07:00:46.031581 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.031779 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":117,"threads_started":1}
I20260812 07:00:46.032109 9728 raft_consensus.cc:348] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.032243 9728 raft_consensus.cc:374] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:46.032290 9728 raft_consensus.cc:729] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9489275ce6a3415b856fce223d4d97c3, State: Initialized, Role: FOLLOWER
I20260812 07:00:46.032459 9728 consensus_queue.cc:260] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.032809 9728 ts_tablet_manager.cc:1428] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 07:00:46.034065 9734 tablet_bootstrap.cc:492] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: No bootstrap required, opened a new log
I20260812 07:00:46.034158 9734 ts_tablet_manager.cc:1397] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 07:00:46.034587 9734 raft_consensus.cc:348] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.034680 9734 raft_consensus.cc:374] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:46.034672 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.033732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.194:50009)
0812 07:00:46.034502 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.034564 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":100,"threads_started":1}
I20260812 07:00:46.034703 9734 raft_consensus.cc:729] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6246dc472d4d4b9f04d5b72d726c90, State: Initialized, Role: FOLLOWER
I20260812 07:00:46.034837 9734 consensus_queue.cc:260] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.035202 9734 ts_tablet_manager.cc:1428] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260812 07:00:46.035212 9068 heartbeater.cc:503] Master 127.27.110.252:42849 was elected leader, sending a full tablet report...
I20260812 07:00:46.036208 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.035754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:51449)
0812 07:00:46.036042 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.036103 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":125,"threads_started":1}
I20260812 07:00:46.036775 9574 heartbeater.cc:503] Master 127.27.110.252:42849 was elected leader, sending a full tablet report...
I20260812 07:00:46.036911 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.036233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:40683)
0812 07:00:46.036784 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.036840 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481}
I20260812 07:00:46.038054 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.037661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:48095)
0812 07:00:46.037787 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.037991 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 07:00:46.241614 9736 raft_consensus.cc:482] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:46.241811 9736 raft_consensus.cc:504] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.242355 9736 leader_election.cc:288] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893), 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819)
I20260812 07:00:46.242844 9220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e952e8de33be4786b8b3a6c0fdcbd2d3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9489275ce6a3415b856fce223d4d97c3" is_pre_election: true
I20260812 07:00:46.243033 9220 raft_consensus.cc:2454] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 0.
I20260812 07:00:46.243016 9491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e952e8de33be4786b8b3a6c0fdcbd2d3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" is_pre_election: true
I20260812 07:00:46.243222 9491 raft_consensus.cc:2454] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 0.
I20260812 07:00:46.243511 8901 leader_election.cc:302] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 676664633f2445849855b82d215f7c7f, 9489275ce6a3415b856fce223d4d97c3; no voters:
I20260812 07:00:46.243793 9736 raft_consensus.cc:2781] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:46.243872 9736 raft_consensus.cc:482] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:46.243932 9736 raft_consensus.cc:3037] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:46.245114 9736 raft_consensus.cc:504] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.245493 9736 leader_election.cc:288] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 election: Requested vote from peers 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893), 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819)
I20260812 07:00:46.245926 9220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e952e8de33be4786b8b3a6c0fdcbd2d3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9489275ce6a3415b856fce223d4d97c3"
I20260812 07:00:46.245923 9491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e952e8de33be4786b8b3a6c0fdcbd2d3" candidate_uuid: "676664633f2445849855b82d215f7c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6246dc472d4d4b9f04d5b72d726c90"
I20260812 07:00:46.246058 9491 raft_consensus.cc:3037] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:46.246083 9220 raft_consensus.cc:3037] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:46.247282 9491 raft_consensus.cc:2454] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 1.
I20260812 07:00:46.247282 9220 raft_consensus.cc:2454] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 676664633f2445849855b82d215f7c7f in term 1.
I20260812 07:00:46.247576 8904 leader_election.cc:302] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6246dc472d4d4b9f04d5b72d726c90, 676664633f2445849855b82d215f7c7f; no voters:
I20260812 07:00:46.247824 9736 raft_consensus.cc:2781] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:46.248026 9736 raft_consensus.cc:686] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 1 LEADER]: Becoming Leader. State: Replica: 676664633f2445849855b82d215f7c7f, State: Running, Role: LEADER
I20260812 07:00:46.248178 9736 consensus_queue.cc:237] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } }
I20260812 07:00:46.248808 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.248540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:42737)
0812 07:00:46.248666 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.248736 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260812 07:00:46.250032 8786 catalog_manager.cc:5495] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 676664633f2445849855b82d215f7c7f (127.27.110.193). New cstate: current_term: 1 leader_uuid: "676664633f2445849855b82d215f7c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "676664633f2445849855b82d215f7c7f" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 45225 } health_report { overall_health: HEALTHY } } }
I20260812 07:00:46.316278 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.313207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.27.110.193:45225)
0812 07:00:46.313353 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:46.313357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:46.313371 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:46.313422 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:46.313425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:46.313511 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:46.313581 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:46.313855 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.314368 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.315205 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:46.315551 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:46.315637 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:46.315862 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:46.315930 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:46.315984 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:46.316116 (+ 132us) server_negotiation.cc:299] Negotiation successful
0812 07:00:46.316166 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:00:46.318707 9220 raft_consensus.cc:1261] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Refusing update from remote peer 676664633f2445849855b82d215f7c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:46.318755 9491 raft_consensus.cc:1261] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Refusing update from remote peer 676664633f2445849855b82d215f7c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:46.319227 9747 consensus_queue.cc:1035] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9489275ce6a3415b856fce223d4d97c3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:46.319416 9736 consensus_queue.cc:1035] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6246dc472d4d4b9f04d5b72d726c90" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 41819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:46.320335 9313 heartbeater.cc:503] Master 127.27.110.252:42849 was elected leader, sending a full tablet report...
I20260812 07:00:46.321085 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.320794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.194:58931)
0812 07:00:46.320945 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.321008 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260812 07:00:46.322794 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.322484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:35025)
0812 07:00:46.322659 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.322713 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 07:00:46.325433 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.325021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:38369)
0812 07:00:46.325301 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.325359 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260812 07:00:46.325735 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.325437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.195:34425)
0812 07:00:46.325617 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.325668 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260812 07:00:46.326045 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.325628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.194:40053)
0812 07:00:46.325907 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.325960 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215}
I20260812 07:00:46.327560 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.326802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:43403 (local address 127.27.110.193:48267)
0812 07:00:46.327437 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.327494 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:43403: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565}
I20260812 07:00:46.331522 8800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:46.331668 8800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:46.335595 8800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83e4b9bd53a44347828fd9b9eb3df233: Sending DeleteTablet for 3 replicas of tablet e952e8de33be4786b8b3a6c0fdcbd2d3
I20260812 07:00:46.336261 9184 tablet_service.cc:1514] Processing DeleteTablet for tablet e952e8de33be4786b8b3a6c0fdcbd2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:46 UTC) from {username='oryx', principal='oryx/127.27.110.252@KRBTEST.COM'} at 127.0.0.1:45470
I20260812 07:00:46.336267 9455 tablet_service.cc:1514] Processing DeleteTablet for tablet e952e8de33be4786b8b3a6c0fdcbd2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:46 UTC) from {username='oryx', principal='oryx/127.27.110.252@KRBTEST.COM'} at 127.0.0.1:51416
I20260812 07:00:46.336385 8972 tablet_service.cc:1514] Processing DeleteTablet for tablet e952e8de33be4786b8b3a6c0fdcbd2d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:46 UTC) from {username='oryx', principal='oryx/127.27.110.252@KRBTEST.COM'} at 127.0.0.1:33668
I20260812 07:00:46.336671 9770 tablet_replica.cc:330] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: stopping tablet replica
I20260812 07:00:46.336747 9774 tablet_replica.cc:330] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: stopping tablet replica
I20260812 07:00:46.336812 9770 raft_consensus.cc:2229] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:46.336855 9774 raft_consensus.cc:2229] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:46.336948 9770 raft_consensus.cc:2258] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:46.337057 9774 raft_consensus.cc:2258] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:46.337196 9773 tablet_replica.cc:330] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: stopping tablet replica
I20260812 07:00:46.337563 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 8738
I20260812 07:00:46.337566 9773 raft_consensus.cc:2229] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:46.337713 9773 raft_consensus.cc:2258] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:46.337765 9774 ts_tablet_manager.cc:1905] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 676664633f2445849855b82d215f7c7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:46.341912 9773 ts_tablet_manager.cc:1905] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:46.344609 9773 ts_tablet_manager.cc:1918] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:46.344702 9773 log.cc:1196] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/e952e8de33be4786b8b3a6c0fdcbd2d3
I20260812 07:00:46.344986 9773 ts_tablet_manager.cc:1939] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 2c6246dc472d4d4b9f04d5b72d726c90: Deleting consensus metadata
I20260812 07:00:46.345674 8775 catalog_manager.cc:4841] TS 2c6246dc472d4d4b9f04d5b72d726c90 (127.27.110.195:41819): tablet e952e8de33be4786b8b3a6c0fdcbd2d3 (table test-table [id=3ddc195103644e57b3f3b02d7069795e]) successfully deleted
I20260812 07:00:46.347637 9770 ts_tablet_manager.cc:1905] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:46.349586 9770 ts_tablet_manager.cc:1918] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:46.349655 9770 log.cc:1196] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/e952e8de33be4786b8b3a6c0fdcbd2d3
I20260812 07:00:46.349967 9770 ts_tablet_manager.cc:1939] T e952e8de33be4786b8b3a6c0fdcbd2d3 P 9489275ce6a3415b856fce223d4d97c3: Deleting consensus metadata
I20260812 07:00:46.350641 8772 catalog_manager.cc:4841] TS 9489275ce6a3415b856fce223d4d97c3 (127.27.110.194:44893): tablet e952e8de33be4786b8b3a6c0fdcbd2d3 (table test-table [id=3ddc195103644e57b3f3b02d7069795e]) successfully deleted
I20260812 07:00:46.352766 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 9083
I20260812 07:00:46.352918 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:46.352630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:45225 (local address 127.0.0.1:33672)
0812 07:00:46.352759 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:46.352813 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:45225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 07:00:46.360702 8773 catalog_manager.cc:4584] TS 676664633f2445849855b82d215f7c7f (127.27.110.193:45225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e952e8de33be4786b8b3a6c0fdcbd2d3: Network error: Client connection negotiation failed: client connection to 127.27.110.193:45225: connect: Connection refused (error 111)
I20260812 07:00:46.372233 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 9326
I20260812 07:00:46.389086 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 8267
I20260812 07:00:46.402117 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 8447
2026-08-12T07:00:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6011 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:46 dist-test-slave-w4v5 krb5kdc[9784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:46 dist-test-slave-w4v5 krb5kdc[9784](info): set up 2 sockets
Aug 12 07:00:46 dist-test-slave-w4v5 krb5kdc[9784](info): commencing operation
krb5kdc: starting...
W20260812 07:00:47.724368 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:47 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab.
Entry for principal oryx/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab.
I20260812 07:00:47.865806 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:33941
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab
--principal=oryx/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:47.981642 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:47.981962 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:47.982048 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:47.986029 9805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:47.986116 9805 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 07:00:47.986151 9805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:47.986191 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:47.986228 9805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:47.986263 9805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:47.991482 9805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:33941
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.254.keytab
--principal=oryx/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:47.992543 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:47.993566 9805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:48.002568 9813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.002583 9818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.002578 9815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:48.003118 9805 server_base.cc:1034] running on GCE node
I20260812 07:00:48.003733 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:48.004873 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:48.006095 9805 hybrid_clock.cc:648] HybridClock initialized: now 1786518048006080 us; error 38 us; skew 500 ppm
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:48.009042 9805 init.cc:377] Logged in from keytab as oryx/127.27.110.254@KRBTEST.COM (short username oryx)
I20260812 07:00:48.009965 9805 webserver.cc:460] Webserver started at http://127.27.110.254:38789/ using document root <none> and password file <none>
I20260812 07:00:48.010572 9805 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:48.010671 9805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:48.010915 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:48.013163 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "8f3f33c024f14587a7132ed5802d8e43"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "fce2b06a0762c0bc5e0b53e526f01f39"
server_key_iv: "ec5676a3b978adde26bccd30787fa054"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.013756 9805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "8f3f33c024f14587a7132ed5802d8e43"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "fce2b06a0762c0bc5e0b53e526f01f39"
server_key_iv: "ec5676a3b978adde26bccd30787fa054"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.017689 9805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:00:48.020242 9827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.021620 9805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:48.021787 9805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "8f3f33c024f14587a7132ed5802d8e43"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "fce2b06a0762c0bc5e0b53e526f01f39"
server_key_iv: "ec5676a3b978adde26bccd30787fa054"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.021910 9805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:48.053555 9805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:48.056905 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:48.057135 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:48.066020 9805 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:33941
I20260812 07:00:48.066045 9936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:33941 every 8 connection(s)
I20260812 07:00:48.067022 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:48.069912 9937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:48.073282 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 9805
I20260812 07:00:48.073433 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:48.073732 28091 external_mini_cluster.cc:1428] Setting key d6c89a402d48ea96742179cf0cda3513
I20260812 07:00:48.075953 9937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Bootstrap starting.
I20260812 07:00:48.078548 9937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:48.079453 9937 log.cc:824] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:48.081722 9937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: No bootstrap required, opened a new log
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:48.086325 9937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f33c024f14587a7132ed5802d8e43" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33941 } }
I20260812 07:00:48.086632 9937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:48.086696 9937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3f33c024f14587a7132ed5802d8e43, State: Initialized, Role: FOLLOWER
I20260812 07:00:48.087206 9937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f33c024f14587a7132ed5802d8e43" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33941 } }
I20260812 07:00:48.087374 9937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:48.087466 9937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:48.087568 9937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:48.088603 9937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f33c024f14587a7132ed5802d8e43" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33941 } }
I20260812 07:00:48.088953 9937 leader_election.cc:302] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f3f33c024f14587a7132ed5802d8e43; no voters:
I20260812 07:00:48.089236 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.075182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.27.110.254:33941)
0812 07:00:48.075616 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.075632 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.075667 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:00:48.076339 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.076357 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.076688 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.076967 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.077920 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.079002 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.080173 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.080675 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.080804 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.083766 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.083773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.083790 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.083825 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.085788 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.086245 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.086250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.086255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.086408 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.086774 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.086797 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.086802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.087370 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.087619 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.088088 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.088388 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260812 07:00:48.089303 9937 leader_election.cc:288] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:48.089380 9944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:48.089568 9944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [term 1 LEADER]: Becoming Leader. State: Replica: 8f3f33c024f14587a7132ed5802d8e43, State: Running, Role: LEADER
I20260812 07:00:48.089860 9944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f33c024f14587a7132ed5802d8e43" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33941 } }
I20260812 07:00:48.090592 9937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:48.091218 9945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f3f33c024f14587a7132ed5802d8e43. Latest consensus state: current_term: 1 leader_uuid: "8f3f33c024f14587a7132ed5802d8e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f33c024f14587a7132ed5802d8e43" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33941 } } }
I20260812 07:00:48.091372 9945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:48.092025 9946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f3f33c024f14587a7132ed5802d8e43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f3f33c024f14587a7132ed5802d8e43" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 33941 } } }
I20260812 07:00:48.092124 9946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43 [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:48.092409 9952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:48.095845 9952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:48.102120 9952 catalog_manager.cc:1347] Generated new cluster ID: b5b5363f2bbe49f4b407c6eeb98356fd
I20260812 07:00:48.102211 9952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:48.107033 9952 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:48.108314 9952 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:48.116583 9952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Generated new TSK 0
I20260812 07:00:48.117429 9952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab.
Entry for principal oryx/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab.
I20260812 07:00:48.147647 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:33941
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:48.265089 9978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:48.265365 9978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:48.265410 9978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:48.269838 9978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:48.269924 9978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:48.269968 9978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:48.270058 9978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:48.275794 9978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:33941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:48.276940 9978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:48.277993 9978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:48.287806 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.287876 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.287876 9997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:48.288105 9978 server_base.cc:1034] running on GCE node
I20260812 07:00:48.288506 9978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:48.289213 9978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:48.290380 9978 hybrid_clock.cc:648] HybridClock initialized: now 1786518048290367 us; error 24 us; skew 500 ppm
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:48.293394 9978 init.cc:377] Logged in from keytab as oryx/127.27.110.193@KRBTEST.COM (short username oryx)
I20260812 07:00:48.294401 9978 webserver.cc:460] Webserver started at http://127.27.110.193:34925/ using document root <none> and password file <none>
I20260812 07:00:48.294977 9978 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:48.295073 9978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:48.295347 9978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:48.297448 9978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "bb8b0722afc22526bfb665ac23f03c42"
server_key_iv: "f127f07fd702df96a563b1e40552476e"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.298040 9978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "bb8b0722afc22526bfb665ac23f03c42"
server_key_iv: "f127f07fd702df96a563b1e40552476e"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.301934 9978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 07:00:48.304510 10008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.305761 9978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:48.305912 9978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "bb8b0722afc22526bfb665ac23f03c42"
server_key_iv: "f127f07fd702df96a563b1e40552476e"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.306027 9978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:48.322649 9978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:48.325961 9978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:48.326174 9978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:48.327839 9978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:48.327924 9978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.327983 9978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:48.328013 9978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:48.339376 9978 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:33189
I20260812 07:00:48.339534 10207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:33189 every 8 connection(s)
I20260812 07:00:48.340410 9978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:48.342502 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.328494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:48045 (local address 127.27.110.254:33941)
0812 07:00:48.328630 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.328634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.329645 (+ 1011us) server_negotiation.cc:407] Connection header received
0812 07:00:48.330610 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.330618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.330681 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.330789 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.332459 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.333296 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.334491 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.334792 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.334929 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.338716 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.338720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.338723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.338753 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.340394 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.340979 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.340983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.340984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.341077 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.341464 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.341467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.341468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.341662 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.341754 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.342261 (+ 507us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.342380 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:00:48.343324 10020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.328996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.27.110.193:48045)
0812 07:00:48.329488 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.329532 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.330380 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.330977 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.330986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.331545 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:48.332247 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.332264 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.333439 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.333443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.334352 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.334363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.335005 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.335012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.335297 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.335904 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:48.335929 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:48.338525 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:48.340532 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.340538 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.340552 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.340823 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.341185 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.341189 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.341191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.341357 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.341766 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:48.341771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:48.341991 (+ 220us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.342145 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.342392 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
I20260812 07:00:48.344718 10210 heartbeater.cc:347] Connected to a master server at 127.27.110.254:33941
I20260812 07:00:48.344887 10210 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:48.344879 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 9978
I20260812 07:00:48.345031 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:48.345417 28091 external_mini_cluster.cc:1428] Setting key 91a12d0885e80f0c959c4f8609da1668
I20260812 07:00:48.345428 10210 heartbeater.cc:511] Master 127.27.110.254:33941 requested a full tablet report, sending...
I20260812 07:00:48.347194 9849 ts_manager.cc:194] Registered new tserver with Master: ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189)
I20260812 07:00:48.348369 9849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:48045
I20260812 07:00:48.357393 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.347246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:55665 (local address 127.27.110.254:33941)
0812 07:00:48.347377 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.347381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.349553 (+ 2172us) server_negotiation.cc:407] Connection header received
0812 07:00:48.349683 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.349688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.349732 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.349817 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.350335 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.351192 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.352461 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.352674 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.352762 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.354182 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.354185 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.354188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.354211 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.355690 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.356214 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.356217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.356218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.356292 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.356595 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.356597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.356598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.356756 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.356829 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.357184 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.357278 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:00:48.357537 10224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.349059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.27.110.193:55665)
0812 07:00:48.349462 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.349474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.349591 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.349836 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.349839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.350089 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:48.350218 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.350225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.351322 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.351329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.352323 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.352338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.352802 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.352809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.352956 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.353517 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:48.353537 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:48.354025 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:48.355810 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.355816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.355820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.356103 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.356391 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.356395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.356397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.356497 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.356861 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:48.356866 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:48.356992 (+ 126us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.357117 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.357293 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for oryx/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab.
Entry for principal oryx/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab.
I20260812 07:00:48.376519 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:33941
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:48.489998 10228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:48.490250 10228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:48.490334 10228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:48.494494 10228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:48.494557 10228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:48.494653 10228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:48.494699 10228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:48.499912 10228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:33941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:48.500970 10228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:48.502003 10228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:48.511677 10238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.511727 10239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.511737 10241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:48.511955 10228 server_base.cc:1034] running on GCE node
I20260812 07:00:48.512406 10228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:48.513047 10228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:48.514263 10228 hybrid_clock.cc:648] HybridClock initialized: now 1786518048514233 us; error 52 us; skew 500 ppm
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:48.517081 10228 init.cc:377] Logged in from keytab as oryx/127.27.110.194@KRBTEST.COM (short username oryx)
I20260812 07:00:48.517933 10228 webserver.cc:460] Webserver started at http://127.27.110.194:38557/ using document root <none> and password file <none>
I20260812 07:00:48.518496 10228 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:48.518620 10228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:48.518848 10228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:48.521001 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "19e02334144a460c9297ddee13a3b37e"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "7bd356299f575827eda0bc8d13bebd7f"
server_key_iv: "14f62ea8a7119dd615c16adff2480710"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.521713 10228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "19e02334144a460c9297ddee13a3b37e"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "7bd356299f575827eda0bc8d13bebd7f"
server_key_iv: "14f62ea8a7119dd615c16adff2480710"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.525547 10228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 07:00:48.528090 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.529366 10228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:48.529489 10228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "19e02334144a460c9297ddee13a3b37e"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "7bd356299f575827eda0bc8d13bebd7f"
server_key_iv: "14f62ea8a7119dd615c16adff2480710"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.529613 10228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:48.544649 10228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:48.547639 10228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:48.547838 10228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:48.549253 10228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:48.549300 10228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.549340 10228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:48.549396 10228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:48.560467 10228 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:36571
I20260812 07:00:48.560554 10446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:36571 every 8 connection(s)
I20260812 07:00:48.561656 10228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:48.563330 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 10228
I20260812 07:00:48.563508 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:48.563881 28091 external_mini_cluster.cc:1428] Setting key 51f97c03b57d720dc78a96a739949755
I20260812 07:00:48.564457 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.550292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:53347 (local address 127.27.110.254:33941)
0812 07:00:48.550422 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.550425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.551537 (+ 1112us) server_negotiation.cc:407] Connection header received
0812 07:00:48.552805 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.552812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.552887 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.552993 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.554416 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.555356 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.557272 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.557504 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.557594 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.560604 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.560608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.560611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.560640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.562246 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.562799 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.562802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.562803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.562884 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.563213 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.563217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.563219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.563415 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.563538 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.564161 (+ 623us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.564286 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 07:00:48.565440 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.550850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.27.110.194:53347)
0812 07:00:48.551370 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.551414 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.552524 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.553166 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.553174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.553581 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:48.554258 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.554283 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.555533 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.555539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.557148 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.557162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.557706 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.557712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.557937 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.558668 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:48.558684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:48.560381 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:48.562390 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.562398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.562413 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.562697 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.562954 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.562955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.562957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.563069 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.563561 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:48.563569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:48.563879 (+ 310us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.564090 (+ 211us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.564418 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":156,"threads_started":1}
I20260812 07:00:48.566664 10449 heartbeater.cc:347] Connected to a master server at 127.27.110.254:33941
I20260812 07:00:48.566799 10449 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:48.567394 10449 heartbeater.cc:511] Master 127.27.110.254:33941 requested a full tablet report, sending...
I20260812 07:00:48.568497 9849 ts_manager.cc:194] Registered new tserver with Master: 19e02334144a460c9297ddee13a3b37e (127.27.110.194:36571)
I20260812 07:00:48.569087 9849 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:53347
WARNING: no policy specified for oryx/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:48.577839 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.568971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:44399 (local address 127.27.110.254:33941)
0812 07:00:48.569095 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.569101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.569724 (+ 623us) server_negotiation.cc:407] Connection header received
0812 07:00:48.569803 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.569806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.569851 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.569939 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.570603 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.571366 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.572690 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.572900 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.572983 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.574325 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.574328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.574330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.574352 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.576091 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.576567 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.576571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.576572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.576660 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.576952 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.576956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.576958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.577183 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.577297 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.577563 (+ 266us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.577687 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:00:48.577857 10459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.569206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.27.110.194:44399)
0812 07:00:48.569619 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.569636 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.569736 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.569979 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.569985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.570227 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:48.570460 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.570473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.571505 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.571512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.572536 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.572549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.573028 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.573035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.573177 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.573691 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:48.573712 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:48.574177 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:48.576239 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.576242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.576244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.576455 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.576759 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.576762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.576764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.576854 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.577286 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:48.577289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:48.577392 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.577506 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.577657 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab.
Entry for principal oryx/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab.
I20260812 07:00:48.595803 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:33941
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:48.712368 10461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:48.712653 10461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:48.712703 10461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:48.716753 10461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:48.716816 10461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:48.716856 10461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 07:00:48.716889 10461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:48.722349 10461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.110.254:33941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:48.723457 10461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:48.724450 10461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:48.734690 10473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.734776 10477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:48.734706 10471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:48.734925 10461 server_base.cc:1034] running on GCE node
I20260812 07:00:48.735488 10461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:48.736133 10461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:48.737298 10461 hybrid_clock.cc:648] HybridClock initialized: now 1786518048737250 us; error 64 us; skew 500 ppm
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:48.740236 10461 init.cc:377] Logged in from keytab as oryx/127.27.110.195@KRBTEST.COM (short username oryx)
I20260812 07:00:48.741209 10461 webserver.cc:460] Webserver started at http://127.27.110.195:45715/ using document root <none> and password file <none>
I20260812 07:00:48.741782 10461 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:48.741878 10461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:48.742115 10461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:48.744194 10461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "59a0bfa35f5044f891741bb14efe5751"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "711cbbe3ed89430d158617367fbc81d8"
server_key_iv: "1f0e43006568d414fe38641e9b976088"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.744805 10461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "59a0bfa35f5044f891741bb14efe5751"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "711cbbe3ed89430d158617367fbc81d8"
server_key_iv: "1f0e43006568d414fe38641e9b976088"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.748703 10461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:48.751195 10489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.752403 10461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:48.752542 10461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "59a0bfa35f5044f891741bb14efe5751"
format_stamp: "Formatted at 2026-08-12 07:00:48 on dist-test-slave-w4v5"
server_key: "711cbbe3ed89430d158617367fbc81d8"
server_key_iv: "1f0e43006568d414fe38641e9b976088"
server_key_version: "encryptionkey@0"
I20260812 07:00:48.752661 10461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:48.767789 10461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:48.770731 10461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:48.770934 10461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:48.772452 10461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:48.772578 10461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:48.772682 10461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:48.772712 10461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:48.784650 10461 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44627
I20260812 07:00:48.784714 10693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44627 every 8 connection(s)
I20260812 07:00:48.785737 10461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:48.787190 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.773498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:45731 (local address 127.27.110.254:33941)
0812 07:00:48.773690 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.773697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.774608 (+ 911us) server_negotiation.cc:407] Connection header received
0812 07:00:48.775866 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.775870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.775942 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.776059 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.777574 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.778481 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.779836 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.780113 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.780231 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.783011 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.783014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.783016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.783064 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.784920 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.785410 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.785414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.785416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.785511 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.785931 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.785934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.785936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.786219 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.786318 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.786902 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.787006 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:00:48.787982 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.773895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.27.110.195:45731)
0812 07:00:48.774412 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.774469 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.775564 (+ 1095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.776253 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.776261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.776695 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:48.777405 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.777423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.778644 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.778649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.779697 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.779708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.780286 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.780291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.780467 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.781062 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:48.781089 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:48.782828 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:48.785029 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.785033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.785043 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.785298 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.785615 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.785617 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.785618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.785757 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.786379 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:48.786387 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:48.786667 (+ 280us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.786827 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.787062 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":149,"threads_started":1}
I20260812 07:00:48.789253 10694 heartbeater.cc:347] Connected to a master server at 127.27.110.254:33941
I20260812 07:00:48.789414 10694 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:48.789913 10694 heartbeater.cc:511] Master 127.27.110.254:33941 requested a full tablet report, sending...
I20260812 07:00:48.790992 9851 ts_manager.cc:194] Registered new tserver with Master: 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:48.791780 9851 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:45731
I20260812 07:00:48.793399 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 10461
I20260812 07:00:48.793543 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:00:48.793872 28091 external_mini_cluster.cc:1428] Setting key 5b3691c9c7a369273fac3d1c5596abf2
I20260812 07:00:48.796108 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:00:48.799137 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.791745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:33377 (local address 127.27.110.254:33941)
0812 07:00:48.791871 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.791878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.792092 (+ 214us) server_negotiation.cc:407] Connection header received
0812 07:00:48.792188 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.792191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.792241 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.792323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.792882 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.793372 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.794219 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.794524 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.794650 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.795908 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.795912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.795915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.795941 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.797393 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.797894 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.797897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.797898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.797972 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.798283 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.798285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.798286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.798440 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.798597 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.798840 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.798950 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:00:48.799154 10701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.791625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.27.110.195:33377)
0812 07:00:48.791998 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.792015 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.792114 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.792328 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.792330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.792568 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:48.792779 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.792787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.793504 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.793507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.794089 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.794098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.794705 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.794710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.794810 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.795335 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:48.795350 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:48.795756 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:48.797508 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.797514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.797519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.797790 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.798067 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:48.798073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:48.798078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:48.798190 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:48.798548 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:48.798552 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:48.798650 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.798772 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.798916 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":20,"thread_start_us":100,"threads_started":1}
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:00:48.822808 9941 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:48.816504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.27.110.254:33941)
0812 07:00:48.816643 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.816647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.816661 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:48.816703 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.816705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.816745 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.816816 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.817555 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.818091 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.818915 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.819214 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.819297 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.822632 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47666: BlockingRecv error: recv got EOF from 127.0.0.1:47666 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 07:00:48 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:48.832515 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.824254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.27.110.254:33941)
0812 07:00:48.824424 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.824428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.824587 (+ 159us) server_negotiation.cc:407] Connection header received
0812 07:00:48.824742 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.824747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.824793 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.824884 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:48.825562 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.826052 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.826914 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.827197 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.827305 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.829486 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:48.829490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:48.829493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:48.829527 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:48.830786 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.831287 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.831292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.831294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.831380 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:48.831667 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:48.831671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:48.831673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:48.831851 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:48.831993 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.832230 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.832346 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 07:00:48.835556 9851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:48.838536 9851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:48.853884 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.848227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.27.110.194:36571)
0812 07:00:48.848606 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.848618 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.849130 (+ 512us) server_negotiation.cc:407] Connection header received
0812 07:00:48.849223 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.849228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.849429 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.849614 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:48.849882 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.851053 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.852232 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.853146 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.853312 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.853488 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.853595 (+ 107us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.853707 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":138,"threads_started":1}
I20260812 07:00:48.853914 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.847272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33189 (local address 127.0.0.1:48006)
0812 07:00:48.847698 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.847755 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.847928 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.849060 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.849067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.849092 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:48.849294 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.849301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.850914 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.850918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.852298 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.852308 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.853459 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.853463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.853594 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.853622 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.853679 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.853747 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":104,"threads_started":1}
I20260812 07:00:48.854236 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.848457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44627 (local address 127.0.0.1:38232)
0812 07:00:48.849712 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.849725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.849822 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.850243 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.850246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.850265 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:48.850347 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.850357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.851555 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.851558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.852647 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.852657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.853827 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.853830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.853917 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.853941 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.854035 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.854120 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187,"thread_start_us":90,"threads_started":1}
I20260812 07:00:48.854624 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.848682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.27.110.195:44627)
0812 07:00:48.849529 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.849563 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.849810 (+ 247us) server_negotiation.cc:407] Connection header received
0812 07:00:48.849948 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.849962 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.850106 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.850255 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:48.850441 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.851416 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.852780 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.853587 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.853710 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.853833 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.854417 (+ 584us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.854515 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":134,"threads_started":1}
I20260812 07:00:48.854764 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.847443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.27.110.193:33189)
0812 07:00:48.847739 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.847747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.847856 (+ 109us) server_negotiation.cc:407] Connection header received
0812 07:00:48.848590 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.848598 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.848844 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.849022 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:48.849426 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.850783 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.852431 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.853299 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.854277 (+ 978us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.854450 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.854504 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.854600 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260812 07:00:48.854884 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.848072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36571 (local address 127.0.0.1:56044)
0812 07:00:48.848903 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.848921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.849016 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.849598 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.849601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.849621 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:48.849713 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.849721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.851202 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.851207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.852098 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.852108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.853282 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.853286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.853393 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.853434 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.853565 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.853709 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":112,"threads_started":1}
I20260812 07:00:48.857744 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7d8e647cd48dca3a77221c8860563 (DEFAULT_TABLE table=test-table [id=d1b6046f12504bbfbab6c9814d407380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:48.857822 10097 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7d8e647cd48dca3a77221c8860563 (DEFAULT_TABLE table=test-table [id=d1b6046f12504bbfbab6c9814d407380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:48.857825 10587 tablet_service.cc:1467] Processing CreateTablet for tablet 2db7d8e647cd48dca3a77221c8860563 (DEFAULT_TABLE table=test-table [id=d1b6046f12504bbfbab6c9814d407380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:48.858815 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7d8e647cd48dca3a77221c8860563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:48.858930 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7d8e647cd48dca3a77221c8860563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:48.858937 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2db7d8e647cd48dca3a77221c8860563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:48.865320 10742 tablet_bootstrap.cc:492] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Bootstrap starting.
I20260812 07:00:48.866696 10743 tablet_bootstrap.cc:492] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Bootstrap starting.
I20260812 07:00:48.868017 10747 tablet_bootstrap.cc:492] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Bootstrap starting.
I20260812 07:00:48.868242 10742 tablet_bootstrap.cc:654] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:48.868896 10743 tablet_bootstrap.cc:654] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:48.869133 10742 log.cc:824] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:48.869724 10743 log.cc:824] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:48.870110 10747 tablet_bootstrap.cc:654] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:48.870994 10747 log.cc:824] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:48.871287 10742 tablet_bootstrap.cc:492] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: No bootstrap required, opened a new log
I20260812 07:00:48.871486 10742 ts_tablet_manager.cc:1397] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:00:48.871766 10743 tablet_bootstrap.cc:492] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: No bootstrap required, opened a new log
I20260812 07:00:48.872016 10743 ts_tablet_manager.cc:1397] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:00:48.873080 10747 tablet_bootstrap.cc:492] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: No bootstrap required, opened a new log
I20260812 07:00:48.873342 10747 ts_tablet_manager.cc:1397] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260812 07:00:48.875587 10742 raft_consensus.cc:348] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.875872 10742 raft_consensus.cc:374] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:48.875972 10742 raft_consensus.cc:729] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Initialized, Role: FOLLOWER
I20260812 07:00:48.876217 10743 raft_consensus.cc:348] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.876463 10743 raft_consensus.cc:374] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:48.876454 10742 consensus_queue.cc:260] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.876549 10743 raft_consensus.cc:729] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2088b725a047f594ae4ab5b7a0f7c7, State: Initialized, Role: FOLLOWER
I20260812 07:00:48.876842 10747 raft_consensus.cc:348] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.877087 10747 raft_consensus.cc:374] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:48.877024 10743 consensus_queue.cc:260] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.877156 10747 raft_consensus.cc:729] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a0bfa35f5044f891741bb14efe5751, State: Initialized, Role: FOLLOWER
I20260812 07:00:48.877589 10742 ts_tablet_manager.cc:1428] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:00:48.877650 10210 heartbeater.cc:503] Master 127.27.110.254:33941 was elected leader, sending a full tablet report...
I20260812 07:00:48.877588 10747 consensus_queue.cc:260] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.877965 10449 heartbeater.cc:503] Master 127.27.110.254:33941 was elected leader, sending a full tablet report...
I20260812 07:00:48.878378 10694 heartbeater.cc:503] Master 127.27.110.254:33941 was elected leader, sending a full tablet report...
I20260812 07:00:48.878646 10747 ts_tablet_manager.cc:1428] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260812 07:00:48.879554 10743 ts_tablet_manager.cc:1428] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:00:48.953598 10755 raft_consensus.cc:482] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:48.953881 10755 raft_consensus.cc:504] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.955284 10755 leader_election.cc:288] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:48.959800 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.955742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57029 (local address 127.27.110.193:33189)
0812 07:00:48.955864 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.955868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.956012 (+ 144us) server_negotiation.cc:407] Connection header received
0812 07:00:48.956080 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.956082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.956123 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.956241 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:48.956645 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.957255 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.958730 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.959218 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.959346 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.959406 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.959597 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.959670 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 07:00:48.959816 10274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.955623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33189 (local address 127.27.110.194:57029)
0812 07:00:48.955847 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.955866 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.955978 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.956371 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.956374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.956396 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:48.956536 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.956547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.957373 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.957378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.958582 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.958591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.959338 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.959342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.959467 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.959486 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.959580 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.959658 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260812 07:00:48.959926 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.956216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57517 (local address 127.27.110.195:44627)
0812 07:00:48.956373 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:48.956377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:48.956477 (+ 100us) server_negotiation.cc:407] Connection header received
0812 07:00:48.956601 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:48.956603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:48.956645 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:48.956744 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:48.957105 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.957660 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.958801 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.959307 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.959445 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.959500 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:48.959706 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 07:00:48.959783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:00:48.959990 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:48.955975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44627 (local address 127.27.110.194:57517)
0812 07:00:48.956381 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:48.956395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:48.956487 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:48.956793 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:48.956797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:48.956817 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:48.956955 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.956960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.957783 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.957790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.958655 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:48.958666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:48.959440 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:48.959444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:48.959554 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:48.959578 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:00:48.959689 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 07:00:48.959776 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":151,"threads_started":1}
I20260812 07:00:48.960731 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7d8e647cd48dca3a77221c8860563" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751" is_pre_election: true
I20260812 07:00:48.960768 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7d8e647cd48dca3a77221c8860563" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" is_pre_election: true
I20260812 07:00:48.961051 10617 raft_consensus.cc:2454] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:48.961083 10134 raft_consensus.cc:2454] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:48.961591 10257 leader_election.cc:302] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, 59a0bfa35f5044f891741bb14efe5751; no voters:
I20260812 07:00:48.961905 10755 raft_consensus.cc:2781] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:48.961987 10755 raft_consensus.cc:482] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:48.962018 10755 raft_consensus.cc:3037] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:48.963127 10755 raft_consensus.cc:504] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.963478 10755 leader_election.cc:288] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Requested vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:48.963917 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7d8e647cd48dca3a77221c8860563" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
I20260812 07:00:48.963981 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2db7d8e647cd48dca3a77221c8860563" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751"
I20260812 07:00:48.964080 10134 raft_consensus.cc:3037] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:48.964107 10617 raft_consensus.cc:3037] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:48.965164 10617 raft_consensus.cc:2454] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:48.965214 10134 raft_consensus.cc:2454] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:48.965507 10257 leader_election.cc:302] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, 59a0bfa35f5044f891741bb14efe5751; no voters:
I20260812 07:00:48.965739 10755 raft_consensus.cc:2781] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:48.965992 10755 raft_consensus.cc:686] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Becoming Leader. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Running, Role: LEADER
I20260812 07:00:48.966369 10755 consensus_queue.cc:237] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:48.969808 9848 catalog_manager.cc:5495] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e02334144a460c9297ddee13a3b37e (127.27.110.194). New cstate: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:49.009827 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.006322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.27.110.194:36571)
0812 07:00:49.006460 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.006464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.006478 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:49.006582 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.006584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.006643 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.006728 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.006966 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.007557 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.008404 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.008628 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.008707 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.008968 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.009469 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.009563 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.009646 (+ 83us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.009713 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:00:49.016232 10134 raft_consensus.cc:1261] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.016615 10617 raft_consensus.cc:1261] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.017016 10761 consensus_queue.cc:1035] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.017269 10755 consensus_queue.cc:1035] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.025058 10768 mvcc.cc:205] Tried to move back new op lower bound from 7317577928762011648 to 7317577928569327616. Current Snapshot: MvccSnapshot[applied={T|T < 7317577928762011648}]
I20260812 07:00:49.027459 10769 mvcc.cc:205] Tried to move back new op lower bound from 7317577928762011648 to 7317577928569327616. Current Snapshot: MvccSnapshot[applied={T|T < 7317577928762011648}]
I20260812 07:00:49.033900 9848 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.034109 9848 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.036628 9848 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Sending DeleteTablet for 3 replicas of tablet 2db7d8e647cd48dca3a77221c8860563
I20260812 07:00:49.037472 10322 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7d8e647cd48dca3a77221c8860563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:56044
I20260812 07:00:49.037647 10097 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7d8e647cd48dca3a77221c8860563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 07:00:49.037988 10587 tablet_service.cc:1514] Processing DeleteTablet for tablet 2db7d8e647cd48dca3a77221c8860563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:38232
I20260812 07:00:49.038230 10787 tablet_replica.cc:330] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: stopping tablet replica
I20260812 07:00:49.038444 10788 tablet_replica.cc:330] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: stopping tablet replica
I20260812 07:00:49.038683 10787 raft_consensus.cc:2229] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.038754 10788 raft_consensus.cc:2229] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.038756 9848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 07:00:49.039096 10788 raft_consensus.cc:2258] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 07:00:49.039198 9848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:49.039386 10787 raft_consensus.cc:2258] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.040048 10780 tablet_replica.cc:330] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: stopping tablet replica
I20260812 07:00:49.040326 10780 raft_consensus.cc:2229] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:49.040681 10780 raft_consensus.cc:2258] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.041222 10787 ts_tablet_manager.cc:1905] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.042311 10788 ts_tablet_manager.cc:1905] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.044255 10787 ts_tablet_manager.cc:1918] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:49.044437 10787 log.cc:1196] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/2db7d8e647cd48dca3a77221c8860563
I20260812 07:00:49.044896 10787 ts_tablet_manager.cc:1939] T 2db7d8e647cd48dca3a77221c8860563 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting consensus metadata
I20260812 07:00:49.045763 10788 ts_tablet_manager.cc:1918] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:49.045861 10788 log.cc:1196] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/2db7d8e647cd48dca3a77221c8860563
I20260812 07:00:49.045892 10097 tablet_service.cc:1467] Processing CreateTablet for tablet e7a6c24becf54762a63912be476887b4 (DEFAULT_TABLE table=test-table [id=b98f9028162a4078b6dc28302ce29d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.046195 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a6c24becf54762a63912be476887b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.046191 10788 ts_tablet_manager.cc:1939] T 2db7d8e647cd48dca3a77221c8860563 P 59a0bfa35f5044f891741bb14efe5751: Deleting consensus metadata
I20260812 07:00:49.046494 9832 catalog_manager.cc:4841] TS ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189): tablet 2db7d8e647cd48dca3a77221c8860563 (table test-table [id=d1b6046f12504bbfbab6c9814d407380]) successfully deleted
I20260812 07:00:49.047381 10587 tablet_service.cc:1467] Processing CreateTablet for tablet e7a6c24becf54762a63912be476887b4 (DEFAULT_TABLE table=test-table [id=b98f9028162a4078b6dc28302ce29d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.047622 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a6c24becf54762a63912be476887b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.048329 10743 tablet_bootstrap.cc:492] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Bootstrap starting.
I20260812 07:00:49.048647 9832 catalog_manager.cc:4841] TS 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627): tablet 2db7d8e647cd48dca3a77221c8860563 (table test-table [id=d1b6046f12504bbfbab6c9814d407380]) successfully deleted
I20260812 07:00:49.049227 10743 tablet_bootstrap.cc:654] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.049683 10747 tablet_bootstrap.cc:492] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Bootstrap starting.
I20260812 07:00:49.049868 10322 tablet_service.cc:1467] Processing CreateTablet for tablet e7a6c24becf54762a63912be476887b4 (DEFAULT_TABLE table=test-table [id=b98f9028162a4078b6dc28302ce29d65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.050134 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a6c24becf54762a63912be476887b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.050575 10747 tablet_bootstrap.cc:654] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.051765 10747 tablet_bootstrap.cc:492] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: No bootstrap required, opened a new log
I20260812 07:00:49.051849 10747 ts_tablet_manager.cc:1397] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260812 07:00:49.051987 10743 tablet_bootstrap.cc:492] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: No bootstrap required, opened a new log
I20260812 07:00:49.052064 10743 ts_tablet_manager.cc:1397] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260812 07:00:49.052342 10747 raft_consensus.cc:348] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.052443 10747 raft_consensus.cc:374] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.052475 10747 raft_consensus.cc:729] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a0bfa35f5044f891741bb14efe5751, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.052472 10742 tablet_bootstrap.cc:492] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Bootstrap starting.
I20260812 07:00:49.052604 10747 consensus_queue.cc:260] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.052644 10743 raft_consensus.cc:348] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.052733 10743 raft_consensus.cc:374] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.052765 10743 raft_consensus.cc:729] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2088b725a047f594ae4ab5b7a0f7c7, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.052892 10747 ts_tablet_manager.cc:1428] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 07:00:49.052902 10743 consensus_queue.cc:260] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.053157 10743 ts_tablet_manager.cc:1428] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:49.053435 10742 tablet_bootstrap.cc:654] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.055426 10780 ts_tablet_manager.cc:1905] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.056746 10742 tablet_bootstrap.cc:492] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: No bootstrap required, opened a new log
I20260812 07:00:49.056854 10742 ts_tablet_manager.cc:1397] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260812 07:00:49.057317 10742 raft_consensus.cc:348] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.057430 10742 raft_consensus.cc:374] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.057471 10742 raft_consensus.cc:729] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.057622 10742 consensus_queue.cc:260] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.057857 10780 ts_tablet_manager.cc:1918] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:49.057964 10780 log.cc:1196] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/2db7d8e647cd48dca3a77221c8860563
I20260812 07:00:49.058296 10780 ts_tablet_manager.cc:1939] T 2db7d8e647cd48dca3a77221c8860563 P 19e02334144a460c9297ddee13a3b37e: Deleting consensus metadata
I20260812 07:00:49.057876 10742 ts_tablet_manager.cc:1428] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:49.059405 9832 catalog_manager.cc:4841] TS 19e02334144a460c9297ddee13a3b37e (127.27.110.194:36571): tablet 2db7d8e647cd48dca3a77221c8860563 (table test-table [id=d1b6046f12504bbfbab6c9814d407380]) successfully deleted
W20260812 07:00:49.062800 10450 tablet.cc:2369] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:00:49.091954 10213 tablet.cc:2369] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:49.147145 10761 raft_consensus.cc:482] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:49.147270 10761 raft_consensus.cc:504] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.147707 10761 leader_election.cc:288] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:49.148272 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6c24becf54762a63912be476887b4" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751" is_pre_election: true
I20260812 07:00:49.148307 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6c24becf54762a63912be476887b4" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" is_pre_election: true
I20260812 07:00:49.148455 10617 raft_consensus.cc:2454] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:49.148469 10134 raft_consensus.cc:2454] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:49.148816 10257 leader_election.cc:302] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, 59a0bfa35f5044f891741bb14efe5751; no voters:
I20260812 07:00:49.149044 10761 raft_consensus.cc:2781] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:49.149127 10761 raft_consensus.cc:482] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:49.149176 10761 raft_consensus.cc:3037] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.150223 10761 raft_consensus.cc:504] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.150566 10761 leader_election.cc:288] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Requested vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:49.150985 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6c24becf54762a63912be476887b4" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751"
I20260812 07:00:49.151043 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a6c24becf54762a63912be476887b4" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
I20260812 07:00:49.151140 10617 raft_consensus.cc:3037] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.151178 10134 raft_consensus.cc:3037] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.152525 10617 raft_consensus.cc:2454] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:49.152530 10134 raft_consensus.cc:2454] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:49.152843 10257 leader_election.cc:302] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.153067 10761 raft_consensus.cc:2781] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:49.153191 10761 raft_consensus.cc:686] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Becoming Leader. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Running, Role: LEADER
I20260812 07:00:49.153329 10761 consensus_queue.cc:237] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.155056 9848 catalog_manager.cc:5495] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e02334144a460c9297ddee13a3b37e (127.27.110.194). New cstate: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } health_report { overall_health: UNKNOWN } } }
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518048, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.254@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:49.209659 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.200862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47670 (local address 127.27.110.254:33941)
0812 07:00:49.201023 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.201030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.201233 (+ 203us) server_negotiation.cc:407] Connection header received
0812 07:00:49.201416 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.201419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.201465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.201591 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:49.202276 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.202802 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.203633 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.203852 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.203958 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.206396 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:49.206399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:49.206401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:49.206425 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:49.207921 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:49.208388 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:49.208391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:49.208392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:49.208475 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:49.208782 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:49.208784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:49.208786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:49.208982 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:49.209059 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.209348 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.209451 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:00:49.209661 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.200806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:33941 (local address 127.0.0.1:47670)
0812 07:00:49.201134 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.201150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.201287 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.201609 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.201612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.202031 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:49.202147 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.202155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.202917 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.202923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.203512 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.203520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.203972 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.203975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.204093 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.204711 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:49.204724 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:49.206255 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:49.208047 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:49.208056 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:49.208061 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:49.208293 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:49.208611 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:49.208613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:49.208615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:49.208699 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:49.209078 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:49.209084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:49.209186 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.209279 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.209388 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20260812 07:00:49.213747 9848 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:47670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 07:00:49.214526 9848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:49.219457 10097 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf30d99723e43c8909e4a29b4132565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7344f1cb9d540c896669d2e0753f200]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:49.219472 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf30d99723e43c8909e4a29b4132565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7344f1cb9d540c896669d2e0753f200]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:49.219529 10587 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf30d99723e43c8909e4a29b4132565 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7344f1cb9d540c896669d2e0753f200]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 07:00:49.219887 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf30d99723e43c8909e4a29b4132565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.219885 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf30d99723e43c8909e4a29b4132565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.219970 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf30d99723e43c8909e4a29b4132565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.222548 10747 tablet_bootstrap.cc:492] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: Bootstrap starting.
I20260812 07:00:49.222764 10742 tablet_bootstrap.cc:492] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: Bootstrap starting.
I20260812 07:00:49.223644 10743 tablet_bootstrap.cc:492] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: Bootstrap starting.
I20260812 07:00:49.223785 10747 tablet_bootstrap.cc:654] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.223801 10742 tablet_bootstrap.cc:654] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.224733 10743 tablet_bootstrap.cc:654] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.225067 10742 tablet_bootstrap.cc:492] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: No bootstrap required, opened a new log
I20260812 07:00:49.225162 10742 ts_tablet_manager.cc:1397] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:49.225358 10747 tablet_bootstrap.cc:492] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: No bootstrap required, opened a new log
I20260812 07:00:49.225438 10747 ts_tablet_manager.cc:1397] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260812 07:00:49.225611 10742 raft_consensus.cc:348] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.225741 10742 raft_consensus.cc:374] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.225780 10742 raft_consensus.cc:729] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.225899 10747 raft_consensus.cc:348] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.225914 10742 consensus_queue.cc:260] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.226018 10747 raft_consensus.cc:374] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.226055 10747 raft_consensus.cc:729] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a0bfa35f5044f891741bb14efe5751, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.226045 10761 raft_consensus.cc:482] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:49.226146 10743 tablet_bootstrap.cc:492] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: No bootstrap required, opened a new log
I20260812 07:00:49.226132 10761 raft_consensus.cc:504] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.226171 10747 consensus_queue.cc:260] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.226222 10743 ts_tablet_manager.cc:1397] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:49.226238 10742 ts_tablet_manager.cc:1428] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:49.226428 10755 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.226450 10757 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.226559 10755 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.226532 10761 leader_election.cc:288] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:49.226586 10757 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.226711 10743 raft_consensus.cc:348] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.226806 10747 ts_tablet_manager.cc:1428] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:49.226820 10743 raft_consensus.cc:374] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.226854 10743 raft_consensus.cc:729] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2088b725a047f594ae4ab5b7a0f7c7, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.226979 10743 consensus_queue.cc:260] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.227279 10743 ts_tablet_manager.cc:1428] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:49.227421 10756 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.227566 10756 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.228000 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf30d99723e43c8909e4a29b4132565" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" is_pre_election: true
I20260812 07:00:49.228157 10134 raft_consensus.cc:2454] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:49.228462 10257 leader_election.cc:302] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.228477 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf30d99723e43c8909e4a29b4132565" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751" is_pre_election: true
I20260812 07:00:49.228605 10617 raft_consensus.cc:2454] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:49.228638 10755 raft_consensus.cc:2781] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:49.228698 10755 raft_consensus.cc:482] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:49.228760 10755 raft_consensus.cc:3037] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.229863 10755 raft_consensus.cc:504] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.230227 10755 leader_election.cc:288] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Requested vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:49.230572 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf30d99723e43c8909e4a29b4132565" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
I20260812 07:00:49.230609 10617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf30d99723e43c8909e4a29b4132565" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751"
I20260812 07:00:49.230729 10134 raft_consensus.cc:3037] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.230734 10617 raft_consensus.cc:3037] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.231901 10617 raft_consensus.cc:2454] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:49.231907 10134 raft_consensus.cc:2454] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:49.232203 10257 leader_election.cc:302] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.232398 10755 raft_consensus.cc:2781] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:49.232546 10755 raft_consensus.cc:686] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Becoming Leader. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Running, Role: LEADER
I20260812 07:00:49.232681 10755 consensus_queue.cc:237] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.233441 10761 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: TxnStatusTablet state changed. Reason: New leader 19e02334144a460c9297ddee13a3b37e. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.233548 10761 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:49.233857 10807 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 07:00:49.234033 10807 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 07:00:49.234454 9849 catalog_manager.cc:5495] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e02334144a460c9297ddee13a3b37e (127.27.110.194). New cstate: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:49.247186 10802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.243479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36571 (local address 127.0.0.1:56052)
0812 07:00:49.243631 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.243650 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.243768 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.244053 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.244055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.244061 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:49.244156 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.244160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.245178 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.245181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.245909 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.245920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.246467 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.246470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.246536 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.246581 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.246874 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.246886 (+ 12us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.246937 (+ 51us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.247005 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 07:00:49.247186 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.243562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.27.110.194:36571)
0812 07:00:49.243690 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.243694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.243710 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:49.243887 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.243891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.243943 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.244027 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.244280 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.245077 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.246018 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.246363 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.246472 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.246710 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.246789 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.246863 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.246974 (+ 111us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.247028 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:00:49.249790 10617 raft_consensus.cc:1261] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.249790 10134 raft_consensus.cc:1261] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.250285 10761 consensus_queue.cc:1035] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.250468 10755 consensus_queue.cc:1035] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.251883 10757 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: TxnStatusTablet state changed. Reason: New leader 19e02334144a460c9297ddee13a3b37e. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.252046 10757 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.252291 10756 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: TxnStatusTablet state changed. Reason: New leader 19e02334144a460c9297ddee13a3b37e. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.252411 10756 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.254951 10761 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.255053 10761 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:49.255551 10757 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.255652 10757 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 59a0bfa35f5044f891741bb14efe5751: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.261246 10761 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.261374 10761 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P 19e02334144a460c9297ddee13a3b37e: This TxnStatusTablet replica's current role is: LEADER
I20260812 07:00:49.262090 10756 tablet_replica.cc:439] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } } }
I20260812 07:00:49.262219 10756 tablet_replica.cc:442] T 3bf30d99723e43c8909e4a29b4132565 P ee2088b725a047f594ae4ab5b7a0f7c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 07:00:49.271757 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.268194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:35539 (local address 127.27.110.194:36571)
0812 07:00:49.268330 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.268335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.268522 (+ 187us) server_negotiation.cc:407] Connection header received
0812 07:00:49.268648 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.268650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.268698 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.268753 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.269011 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.269498 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.270705 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.270904 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.270979 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.271313 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.271378 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.271435 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.271554 (+ 119us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.271604 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":55}
I20260812 07:00:49.271806 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.268126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36571 (local address 127.27.110.194:35539)
0812 07:00:49.268436 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.268450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.268558 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.268788 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.268791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.268798 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:49.268906 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.268912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.269601 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.269604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.270581 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.270593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.271010 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.271014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.271120 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.271179 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.271462 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.271469 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.271537 (+ 68us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.271621 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":127,"threads_started":1}
I20260812 07:00:49.278915 10617 raft_consensus.cc:1261] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.278993 10134 raft_consensus.cc:1261] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.279361 10761 consensus_queue.cc:1035] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.279534 10774 consensus_queue.cc:1035] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.282522 10767 mvcc.cc:205] Tried to move back new op lower bound from 7317577929842614272 to 7317577929333465088. Current Snapshot: MvccSnapshot[applied={T|T < 7317577929842614272}]
I20260812 07:00:49.321388 9849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.321569 9849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.323705 9849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Sending DeleteTablet for 3 replicas of tablet e7a6c24becf54762a63912be476887b4
I20260812 07:00:49.324257 10097 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a6c24becf54762a63912be476887b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 07:00:49.324388 10587 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a6c24becf54762a63912be476887b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:38232
I20260812 07:00:49.324440 10787 tablet_replica.cc:330] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: stopping tablet replica
I20260812 07:00:49.324549 10787 raft_consensus.cc:2229] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.324584 10788 tablet_replica.cc:330] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: stopping tablet replica
I20260812 07:00:49.324666 10787 raft_consensus.cc:2258] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.324735 10788 raft_consensus.cc:2229] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.324856 10788 raft_consensus.cc:2258] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.325028 10326 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a6c24becf54762a63912be476887b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:56044
I20260812 07:00:49.325238 10780 tablet_replica.cc:330] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: stopping tablet replica
I20260812 07:00:49.325358 10780 raft_consensus.cc:2229] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:49.325488 10787 ts_tablet_manager.cc:1905] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.325580 10780 raft_consensus.cc:2258] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.326231 10780 ts_tablet_manager.cc:1905] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.328425 10787 ts_tablet_manager.cc:1918] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:49.328450 10780 ts_tablet_manager.cc:1918] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:49.328508 10787 log.cc:1196] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/e7a6c24becf54762a63912be476887b4
I20260812 07:00:49.328539 10780 log.cc:1196] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/e7a6c24becf54762a63912be476887b4
I20260812 07:00:49.328799 10780 ts_tablet_manager.cc:1939] T e7a6c24becf54762a63912be476887b4 P 19e02334144a460c9297ddee13a3b37e: Deleting consensus metadata
I20260812 07:00:49.328801 10787 ts_tablet_manager.cc:1939] T e7a6c24becf54762a63912be476887b4 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting consensus metadata
I20260812 07:00:49.329432 9832 catalog_manager.cc:4841] TS ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189): tablet e7a6c24becf54762a63912be476887b4 (table test-table [id=b98f9028162a4078b6dc28302ce29d65]) successfully deleted
I20260812 07:00:49.329599 9832 catalog_manager.cc:4841] TS 19e02334144a460c9297ddee13a3b37e (127.27.110.194:36571): tablet e7a6c24becf54762a63912be476887b4 (table test-table [id=b98f9028162a4078b6dc28302ce29d65]) successfully deleted
I20260812 07:00:49.330544 10788 ts_tablet_manager.cc:1905] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.333868 10788 ts_tablet_manager.cc:1918] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:49.333964 10788 log.cc:1196] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/e7a6c24becf54762a63912be476887b4
I20260812 07:00:49.334218 10788 ts_tablet_manager.cc:1939] T e7a6c24becf54762a63912be476887b4 P 59a0bfa35f5044f891741bb14efe5751: Deleting consensus metadata
I20260812 07:00:49.334877 9832 catalog_manager.cc:4841] TS 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627): tablet e7a6c24becf54762a63912be476887b4 (table test-table [id=b98f9028162a4078b6dc28302ce29d65]) successfully deleted
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[9784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:00:49.365073 9941 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:49.357136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.27.110.254:33941)
0812 07:00:49.357291 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.357296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.357591 (+ 295us) server_negotiation.cc:407] Connection header received
0812 07:00:49.357723 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.357727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.357778 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.357855 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:49.358764 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.359597 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.360819 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.361167 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.361274 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.364954 (+ 3680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47676: BlockingRecv error: recv got EOF from 127.0.0.1:47676 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[9784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM
I20260812 07:00:49.375476 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.366924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47690 (local address 127.27.110.254:33941)
0812 07:00:49.367066 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.367089 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.367277 (+ 188us) server_negotiation.cc:407] Connection header received
0812 07:00:49.367422 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.367425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.367474 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.367562 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:49.368229 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.368732 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.369525 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.369748 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.369859 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.372206 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:49.372210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:49.372214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:49.372243 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:49.373786 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:49.374216 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:49.374222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:49.374228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:49.374305 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:49.374590 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:49.374593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:49.374595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:49.374780 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:49.374926 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.375193 (+ 267us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.375312 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:00:49.378290 9849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:00:49.378762 9849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:49.383810 10097 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5645914a324952aad8d4959aacec04 (DEFAULT_TABLE table=test-table [id=6bd547c26a61464d84c398c433b0a584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.383838 10326 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5645914a324952aad8d4959aacec04 (DEFAULT_TABLE table=test-table [id=6bd547c26a61464d84c398c433b0a584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.384130 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5645914a324952aad8d4959aacec04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.384156 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5645914a324952aad8d4959aacec04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.386533 10743 tablet_bootstrap.cc:492] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Bootstrap starting.
I20260812 07:00:49.386577 10742 tablet_bootstrap.cc:492] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Bootstrap starting.
I20260812 07:00:49.387339 10743 tablet_bootstrap.cc:654] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.387584 10742 tablet_bootstrap.cc:654] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.388517 10743 tablet_bootstrap.cc:492] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: No bootstrap required, opened a new log
I20260812 07:00:49.388609 10743 ts_tablet_manager.cc:1397] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:00:49.388806 10587 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5645914a324952aad8d4959aacec04 (DEFAULT_TABLE table=test-table [id=6bd547c26a61464d84c398c433b0a584]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.389086 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5645914a324952aad8d4959aacec04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.389091 10743 raft_consensus.cc:348] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.389192 10743 raft_consensus.cc:374] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.389225 10743 raft_consensus.cc:729] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2088b725a047f594ae4ab5b7a0f7c7, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.389351 10743 consensus_queue.cc:260] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.389643 10743 ts_tablet_manager.cc:1428] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:49.391239 10742 tablet_bootstrap.cc:492] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: No bootstrap required, opened a new log
I20260812 07:00:49.391328 10742 ts_tablet_manager.cc:1397] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 07:00:49.391652 10747 tablet_bootstrap.cc:492] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Bootstrap starting.
I20260812 07:00:49.391816 10742 raft_consensus.cc:348] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.391916 10742 raft_consensus.cc:374] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.391948 10742 raft_consensus.cc:729] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.392076 10742 consensus_queue.cc:260] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.392333 10742 ts_tablet_manager.cc:1428] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 07:00:49.392587 10747 tablet_bootstrap.cc:654] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.393967 10747 tablet_bootstrap.cc:492] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: No bootstrap required, opened a new log
I20260812 07:00:49.394064 10747 ts_tablet_manager.cc:1397] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260812 07:00:49.394555 10747 raft_consensus.cc:348] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.394665 10747 raft_consensus.cc:374] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.394688 10747 raft_consensus.cc:729] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a0bfa35f5044f891741bb14efe5751, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.394816 10747 consensus_queue.cc:260] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.395112 10747 ts_tablet_manager.cc:1428] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260812 07:00:49.446733 10761 raft_consensus.cc:482] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:49.446920 10761 raft_consensus.cc:504] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.447378 10761 leader_election.cc:288] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:49.447878 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5645914a324952aad8d4959aacec04" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" is_pre_election: true
I20260812 07:00:49.447950 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5645914a324952aad8d4959aacec04" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751" is_pre_election: true
I20260812 07:00:49.448053 10134 raft_consensus.cc:2454] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:49.448132 10615 raft_consensus.cc:2454] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 0.
I20260812 07:00:49.448374 10257 leader_election.cc:302] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.448578 10761 raft_consensus.cc:2781] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:49.448639 10761 raft_consensus.cc:482] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:49.448680 10761 raft_consensus.cc:3037] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.449702 10761 raft_consensus.cc:504] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.450013 10761 leader_election.cc:288] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Requested vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627)
I20260812 07:00:49.450413 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5645914a324952aad8d4959aacec04" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
I20260812 07:00:49.450484 10615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5645914a324952aad8d4959aacec04" candidate_uuid: "19e02334144a460c9297ddee13a3b37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59a0bfa35f5044f891741bb14efe5751"
I20260812 07:00:49.450556 10134 raft_consensus.cc:3037] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.450616 10615 raft_consensus.cc:3037] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.451972 10134 raft_consensus.cc:2454] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:49.452008 10615 raft_consensus.cc:2454] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19e02334144a460c9297ddee13a3b37e in term 1.
I20260812 07:00:49.452303 10257 leader_election.cc:302] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19e02334144a460c9297ddee13a3b37e, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.452469 10761 raft_consensus.cc:2781] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:49.452570 10761 raft_consensus.cc:686] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Becoming Leader. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Running, Role: LEADER
I20260812 07:00:49.452718 10761 consensus_queue.cc:237] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.454366 9849 catalog_manager.cc:5495] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e reported cstate change: term changed from 0 to 1, leader changed from <none> to 19e02334144a460c9297ddee13a3b37e (127.27.110.194). New cstate: current_term: 1 leader_uuid: "19e02334144a460c9297ddee13a3b37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } health_report { overall_health: UNKNOWN } } }
I20260812 07:00:49.470966 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.467896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.27.110.194:36571)
0812 07:00:49.468017 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.468020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.468034 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:49.468156 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.468158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.468203 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.468261 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.468477 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.469008 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.469998 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.470207 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.470283 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.470540 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.470606 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.470683 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.470821 (+ 138us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.470864 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260812 07:00:49.472954 10134 raft_consensus.cc:1261] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.472954 10615 raft_consensus.cc:1261] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Refusing update from remote peer 19e02334144a460c9297ddee13a3b37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.473404 10774 consensus_queue.cc:1035] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.473553 10761 consensus_queue.cc:1035] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.481176 9849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.481302 9849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.483119 9849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Sending DeleteTablet for 3 replicas of tablet 0a5645914a324952aad8d4959aacec04
I20260812 07:00:49.483685 10097 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5645914a324952aad8d4959aacec04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 07:00:49.483796 10587 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5645914a324952aad8d4959aacec04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:38232
I20260812 07:00:49.483870 10787 tablet_replica.cc:330] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: stopping tablet replica
I20260812 07:00:49.483911 10326 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5645914a324952aad8d4959aacec04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:56044
I20260812 07:00:49.483984 10787 raft_consensus.cc:2229] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.484040 10788 tablet_replica.cc:330] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: stopping tablet replica
I20260812 07:00:49.484123 10787 raft_consensus.cc:2258] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.484160 10780 tablet_replica.cc:330] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: stopping tablet replica
I20260812 07:00:49.484162 10788 raft_consensus.cc:2229] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.484263 10780 raft_consensus.cc:2229] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:49.484297 10788 raft_consensus.cc:2258] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.484431 10780 raft_consensus.cc:2258] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.484719 9849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 07:00:49.484831 10787 ts_tablet_manager.cc:1905] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 07:00:49.485121 9849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:00:49.485320 10788 ts_tablet_manager.cc:1905] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.486233 10780 ts_tablet_manager.cc:1905] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.487453 10787 ts_tablet_manager.cc:1918] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:49.487540 10787 log.cc:1196] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/0a5645914a324952aad8d4959aacec04
I20260812 07:00:49.487855 10787 ts_tablet_manager.cc:1939] T 0a5645914a324952aad8d4959aacec04 P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting consensus metadata
I20260812 07:00:49.487915 10788 ts_tablet_manager.cc:1918] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:49.488006 10788 log.cc:1196] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/0a5645914a324952aad8d4959aacec04
I20260812 07:00:49.488278 10788 ts_tablet_manager.cc:1939] T 0a5645914a324952aad8d4959aacec04 P 59a0bfa35f5044f891741bb14efe5751: Deleting consensus metadata
I20260812 07:00:49.488539 9832 catalog_manager.cc:4841] TS ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189): tablet 0a5645914a324952aad8d4959aacec04 (table test-table [id=6bd547c26a61464d84c398c433b0a584]) successfully deleted
I20260812 07:00:49.488902 9832 catalog_manager.cc:4841] TS 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627): tablet 0a5645914a324952aad8d4959aacec04 (table test-table [id=6bd547c26a61464d84c398c433b0a584]) successfully deleted
I20260812 07:00:49.489337 10780 ts_tablet_manager.cc:1918] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:00:49.489425 10780 log.cc:1196] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/0a5645914a324952aad8d4959aacec04
I20260812 07:00:49.489682 10780 ts_tablet_manager.cc:1939] T 0a5645914a324952aad8d4959aacec04 P 19e02334144a460c9297ddee13a3b37e: Deleting consensus metadata
I20260812 07:00:49.490481 9832 catalog_manager.cc:4841] TS 19e02334144a460c9297ddee13a3b37e (127.27.110.194:36571): tablet 0a5645914a324952aad8d4959aacec04 (table test-table [id=6bd547c26a61464d84c398c433b0a584]) successfully deleted
I20260812 07:00:49.491784 10097 tablet_service.cc:1467] Processing CreateTablet for tablet 07eb52fc350042faaab2675a961e32af (DEFAULT_TABLE table=test-table [id=3b4ce7ad53264610ad8741a087c10726]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.491780 10326 tablet_service.cc:1467] Processing CreateTablet for tablet 07eb52fc350042faaab2675a961e32af (DEFAULT_TABLE table=test-table [id=3b4ce7ad53264610ad8741a087c10726]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.492045 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07eb52fc350042faaab2675a961e32af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.492045 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07eb52fc350042faaab2675a961e32af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.494236 10743 tablet_bootstrap.cc:492] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: Bootstrap starting.
I20260812 07:00:49.494268 10742 tablet_bootstrap.cc:492] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: Bootstrap starting.
I20260812 07:00:49.495128 10742 tablet_bootstrap.cc:654] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.495131 10743 tablet_bootstrap.cc:654] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.495887 10587 tablet_service.cc:1467] Processing CreateTablet for tablet 07eb52fc350042faaab2675a961e32af (DEFAULT_TABLE table=test-table [id=3b4ce7ad53264610ad8741a087c10726]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:00:49.496130 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07eb52fc350042faaab2675a961e32af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:49.496259 10742 tablet_bootstrap.cc:492] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: No bootstrap required, opened a new log
I20260812 07:00:49.496315 10743 tablet_bootstrap.cc:492] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: No bootstrap required, opened a new log
I20260812 07:00:49.496340 10742 ts_tablet_manager.cc:1397] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:49.496382 10743 ts_tablet_manager.cc:1397] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260812 07:00:49.496738 10742 raft_consensus.cc:348] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.496852 10742 raft_consensus.cc:374] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.496809 10743 raft_consensus.cc:348] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.496884 10742 raft_consensus.cc:729] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e02334144a460c9297ddee13a3b37e, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.496906 10743 raft_consensus.cc:374] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.496942 10743 raft_consensus.cc:729] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2088b725a047f594ae4ab5b7a0f7c7, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.497022 10742 consensus_queue.cc:260] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.497056 10743 consensus_queue.cc:260] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.497305 10742 ts_tablet_manager.cc:1428] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260812 07:00:49.497328 10743 ts_tablet_manager.cc:1428] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260812 07:00:49.498266 10747 tablet_bootstrap.cc:492] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Bootstrap starting.
I20260812 07:00:49.499150 10747 tablet_bootstrap.cc:654] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:49.500439 10747 tablet_bootstrap.cc:492] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: No bootstrap required, opened a new log
I20260812 07:00:49.500531 10747 ts_tablet_manager.cc:1397] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:49.500967 10747 raft_consensus.cc:348] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.501106 10747 raft_consensus.cc:374] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:49.501150 10747 raft_consensus.cc:729] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a0bfa35f5044f891741bb14efe5751, State: Initialized, Role: FOLLOWER
I20260812 07:00:49.501309 10747 consensus_queue.cc:260] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.501601 10747 ts_tablet_manager.cc:1428] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260812 07:00:49.538497 10696 tablet.cc:2369] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:00:49.598882 10757 raft_consensus.cc:482] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:00:49.599025 10757 raft_consensus.cc:504] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.600116 10757 leader_election.cc:288] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 19e02334144a460c9297ddee13a3b37e (127.27.110.194:36571)
I20260812 07:00:49.605083 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.600936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:48645 (local address 127.27.110.194:36571)
0812 07:00:49.601098 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.601102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.601280 (+ 178us) server_negotiation.cc:407] Connection header received
0812 07:00:49.601372 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.601376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.601432 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.601508 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:49.601868 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.602672 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.604017 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.604432 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.604533 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.604615 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.604878 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.604954 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:00:49.605139 10876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.600468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:39127 (local address 127.27.110.193:33189)
0812 07:00:49.600789 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.600794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.600812 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:00:49.601301 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.601307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.601372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.601476 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:00:49.601869 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.602672 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.604044 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.604465 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.604633 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.604690 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.604906 (+ 216us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.604986 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260812 07:00:49.605130 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.600810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36571 (local address 127.27.110.195:48645)
0812 07:00:49.601187 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.601202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.601290 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.601527 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.601529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.601540 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:49.601738 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.601746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.602823 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.602826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.603871 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.603882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.604566 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.604570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.604709 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.604754 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.604826 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.604935 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":89,"thread_start_us":75,"threads_started":1}
I20260812 07:00:49.605144 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.600368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33189 (local address 127.27.110.195:39127)
0812 07:00:49.600699 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.600714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.601135 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.601514 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.601517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.601535 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:00:49.601738 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.601746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.602805 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.602811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.603875 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.603887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.604625 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.604632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.604754 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.604773 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.604870 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.604950 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":148,"thread_start_us":99,"threads_started":1}
I20260812 07:00:49.605656 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eb52fc350042faaab2675a961e32af" candidate_uuid: "59a0bfa35f5044f891741bb14efe5751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" is_pre_election: true
I20260812 07:00:49.605830 10134 raft_consensus.cc:2454] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59a0bfa35f5044f891741bb14efe5751 in term 0.
I20260812 07:00:49.605850 10364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eb52fc350042faaab2675a961e32af" candidate_uuid: "59a0bfa35f5044f891741bb14efe5751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19e02334144a460c9297ddee13a3b37e" is_pre_election: true
I20260812 07:00:49.606108 10364 raft_consensus.cc:2454] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59a0bfa35f5044f891741bb14efe5751 in term 0.
I20260812 07:00:49.606262 10493 leader_election.cc:302] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a0bfa35f5044f891741bb14efe5751, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.606595 10757 raft_consensus.cc:2781] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:00:49.606660 10757 raft_consensus.cc:482] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:00:49.606688 10757 raft_consensus.cc:3037] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.607838 10757 raft_consensus.cc:504] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.608182 10757 leader_election.cc:288] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [CANDIDATE]: Term 1 election: Requested vote from peers ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189), 19e02334144a460c9297ddee13a3b37e (127.27.110.194:36571)
I20260812 07:00:49.608595 10134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eb52fc350042faaab2675a961e32af" candidate_uuid: "59a0bfa35f5044f891741bb14efe5751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7"
I20260812 07:00:49.608621 10364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07eb52fc350042faaab2675a961e32af" candidate_uuid: "59a0bfa35f5044f891741bb14efe5751" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19e02334144a460c9297ddee13a3b37e"
I20260812 07:00:49.608739 10134 raft_consensus.cc:3037] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.608772 10364 raft_consensus.cc:3037] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:49.609758 10134 raft_consensus.cc:2454] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59a0bfa35f5044f891741bb14efe5751 in term 1.
I20260812 07:00:49.609935 10364 raft_consensus.cc:2454] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59a0bfa35f5044f891741bb14efe5751 in term 1.
I20260812 07:00:49.610080 10493 leader_election.cc:302] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59a0bfa35f5044f891741bb14efe5751, ee2088b725a047f594ae4ab5b7a0f7c7; no voters:
I20260812 07:00:49.610309 10757 raft_consensus.cc:2781] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:49.610544 10757 raft_consensus.cc:686] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 1 LEADER]: Becoming Leader. State: Replica: 59a0bfa35f5044f891741bb14efe5751, State: Running, Role: LEADER
I20260812 07:00:49.610807 10757 consensus_queue.cc:237] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } }
I20260812 07:00:49.613200 9849 catalog_manager.cc:5495] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195). New cstate: current_term: 1 leader_uuid: "59a0bfa35f5044f891741bb14efe5751" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59a0bfa35f5044f891741bb14efe5751" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44627 } health_report { overall_health: HEALTHY } } }
I20260812 07:00:49.655285 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.651824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.27.110.195:44627)
0812 07:00:49.652091 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.652098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.652110 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:00:49.652185 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.652187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.652238 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.652347 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.652556 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.653087 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.653862 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.654093 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.654169 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.654437 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.654900 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.654990 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.655046 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.655155 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260812 07:00:49.665488 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.662058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:43285 (local address 127.27.110.194:36571)
0812 07:00:49.662212 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.662216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.662356 (+ 140us) server_negotiation.cc:407] Connection header received
0812 07:00:49.662513 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.662517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.662570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.662655 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.662920 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.663441 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.664333 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.664560 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.664675 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.665019 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.665088 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.665172 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.665296 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.665349 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 07:00:49.665518 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.661899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36571 (local address 127.27.110.195:43285)
0812 07:00:49.662269 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.662282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.662401 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.662668 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.662671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.662680 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:49.662807 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.662813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.663571 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.663574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.664209 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.664218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.664706 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.664710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.664819 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.664861 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.665178 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.665201 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.665272 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.665353 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260812 07:00:49.671892 10134 raft_consensus.cc:1261] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 59a0bfa35f5044f891741bb14efe5751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.672075 10364 raft_consensus.cc:1261] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Refusing update from remote peer 59a0bfa35f5044f891741bb14efe5751: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:00:49.672408 10757 consensus_queue.cc:1035] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2088b725a047f594ae4ab5b7a0f7c7" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 33189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.672636 10878 consensus_queue.cc:1035] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19e02334144a460c9297ddee13a3b37e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:00:49.690722 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.687313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:48015 (local address 127.27.110.195:44627)
0812 07:00:49.687422 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:49.687429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:49.687445 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:00:49.687591 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:49.687594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:49.687645 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:49.687722 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:00:49.688005 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.688608 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.689528 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.689751 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.689830 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.690131 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.690217 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.690306 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:49.690496 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 07:00:49.690560 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 07:00:49.690742 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.687228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44627 (local address 127.27.110.194:48015)
0812 07:00:49.687384 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.687400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:49.687488 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:49.687754 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:49.687757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:49.687767 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 07:00:49.687882 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.687888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.688713 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.688716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.689418 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:49.689427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:49.689873 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:49.689876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:49.689964 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:49.690000 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 07:00:49.690349 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 07:00:49.690356 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:00:49.690464 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 07:00:49.690576 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260812 07:00:49.714318 9851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.714500 9851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:00:49.716749 9851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f3f33c024f14587a7132ed5802d8e43: Sending DeleteTablet for 3 replicas of tablet 07eb52fc350042faaab2675a961e32af
I20260812 07:00:49.717418 10097 tablet_service.cc:1514] Processing DeleteTablet for tablet 07eb52fc350042faaab2675a961e32af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:48006
I20260812 07:00:49.717628 10787 tablet_replica.cc:330] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: stopping tablet replica
I20260812 07:00:49.717648 10587 tablet_service.cc:1514] Processing DeleteTablet for tablet 07eb52fc350042faaab2675a961e32af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:38232
I20260812 07:00:49.717757 10787 raft_consensus.cc:2229] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.717855 10788 tablet_replica.cc:330] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: stopping tablet replica
I20260812 07:00:49.717882 10787 raft_consensus.cc:2258] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.717981 10788 raft_consensus.cc:2229] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:00:49.718299 10788 raft_consensus.cc:2258] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.718429 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 9978
I20260812 07:00:49.718753 10787 ts_tablet_manager.cc:1905] T 07eb52fc350042faaab2675a961e32af P ee2088b725a047f594ae4ab5b7a0f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.722308 10326 tablet_service.cc:1514] Processing DeleteTablet for tablet 07eb52fc350042faaab2675a961e32af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:00:49 UTC) from {username='oryx', principal='oryx/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:56044
I20260812 07:00:49.722517 10780 tablet_replica.cc:330] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: stopping tablet replica
I20260812 07:00:49.722630 10780 raft_consensus.cc:2229] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:00:49.722764 10780 raft_consensus.cc:2258] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:00:49.726899 10788 ts_tablet_manager.cc:1905] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:00:49.729550 10788 ts_tablet_manager.cc:1918] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 07:00:49.729642 10788 log.cc:1196] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/07eb52fc350042faaab2675a961e32af
I20260812 07:00:49.729916 10788 ts_tablet_manager.cc:1939] T 07eb52fc350042faaab2675a961e32af P 59a0bfa35f5044f891741bb14efe5751: Deleting consensus metadata
I20260812 07:00:49.730783 9832 catalog_manager.cc:4841] TS 59a0bfa35f5044f891741bb14efe5751 (127.27.110.195:44627): tablet 07eb52fc350042faaab2675a961e32af (table test-table [id=3b4ce7ad53264610ad8741a087c10726]) successfully deleted
I20260812 07:00:49.734205 10780 ts_tablet_manager.cc:1905] T 07eb52fc350042faaab2675a961e32af P 19e02334144a460c9297ddee13a3b37e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 07:00:49.734861 9832 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:33189 (error 108)
I20260812 07:00:49.734997 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 10228
I20260812 07:00:49.745288 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.738719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:33189 (local address 127.0.0.1:48022)
0812 07:00:49.745016 (+ 6297us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.745126 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:33189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6176,"thread_start_us":67,"threads_started":1}
W20260812 07:00:49.745451 9832 catalog_manager.cc:4584] TS ee2088b725a047f594ae4ab5b7a0f7c7 (127.27.110.193:33189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 07eb52fc350042faaab2675a961e32af: Network error: Client connection negotiation failed: client connection to 127.27.110.193:33189: connect: Connection refused (error 111)
I20260812 07:00:49.752588 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 10461
I20260812 07:00:49.760587 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:49.752407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36571 (local address 127.0.0.1:56072)
0812 07:00:49.760363 (+ 7956us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:49.760454 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.194:36571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7829}
I20260812 07:00:49.770216 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 9805
2026-08-12T07:00:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3361 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[10901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[10901](info): set up 2 sockets
Aug 12 07:00:49 dist-test-slave-w4v5 krb5kdc[10901](info): commencing operation
krb5kdc: starting...
W20260812 07:00:51.087380 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:51 dist-test-slave-w4v5 krb5kdc[10901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:51.215742 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:38419
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:38419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:51.331133 10917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:51.331410 10917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:51.331451 10917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:51.335449 10917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:51.335502 10917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:51.335527 10917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:51.335544 10917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:51.335559 10917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:51.340695 10917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:38419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:38419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:51.341698 10917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:51.342651 10917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:51.351233 10927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:51.351243 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:51.351253 10917 server_base.cc:1034] running on GCE node
W20260812 07:00:51.351469 10928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:51.352097 10917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:51.353231 10917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:51.354405 10917 hybrid_clock.cc:648] HybridClock initialized: now 1786518051354393 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T07:00:51Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1585 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:00:51 dist-test-slave-w4v5 krb5kdc[10939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:00:51 dist-test-slave-w4v5 krb5kdc[10939](info): set up 2 sockets
Aug 12 07:00:51 dist-test-slave-w4v5 krb5kdc[10939](info): commencing operation
krb5kdc: starting...
W20260812 07:00:52.663024 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:00:52 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:00:52.795433 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36415
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:52.909198 10967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:52.909477 10967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:52.909518 10967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:52.913647 10967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:52.913709 10967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:52.913729 10967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:52.913748 10967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:52.913763 10967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:52.919015 10967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:52.920136 10967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:52.921286 10967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:52.930742 10976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:52.930749 10977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:52.930749 10980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:52.930871 10967 server_base.cc:1034] running on GCE node
I20260812 07:00:52.931674 10967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:52.932801 10967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:52.933995 10967 hybrid_clock.cc:648] HybridClock initialized: now 1786518052933965 us; error 47 us; skew 500 ppm
Aug 12 07:00:52 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518052, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:52.936911 10967 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:52.937865 10967 webserver.cc:460] Webserver started at http://127.27.110.254:45873/ using document root <none> and password file <none>
I20260812 07:00:52.938400 10967 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:52.938460 10967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:52.938627 10967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:52.940704 10967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "80152bdbc7f240e6871e1619ab4bb9ad"
format_stamp: "Formatted at 2026-08-12 07:00:52 on dist-test-slave-w4v5"
server_key: "ec81f4ed875d0744671afa04ea282467"
server_key_iv: "30ad260aabba82ddd7ab7ae0a6fa1205"
server_key_version: "encryptionkey@0"
I20260812 07:00:52.941279 10967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "80152bdbc7f240e6871e1619ab4bb9ad"
format_stamp: "Formatted at 2026-08-12 07:00:52 on dist-test-slave-w4v5"
server_key: "ec81f4ed875d0744671afa04ea282467"
server_key_iv: "30ad260aabba82ddd7ab7ae0a6fa1205"
server_key_version: "encryptionkey@0"
I20260812 07:00:52.945128 10967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:52.947623 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:52.948930 10967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:52.949036 10967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "80152bdbc7f240e6871e1619ab4bb9ad"
format_stamp: "Formatted at 2026-08-12 07:00:52 on dist-test-slave-w4v5"
server_key: "ec81f4ed875d0744671afa04ea282467"
server_key_iv: "30ad260aabba82ddd7ab7ae0a6fa1205"
server_key_version: "encryptionkey@0"
I20260812 07:00:52.949127 10967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:52.971500 10967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:52.974684 10967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:52.974845 10967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:52.982239 10967 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36415
I20260812 07:00:52.982265 11084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36415 every 8 connection(s)
I20260812 07:00:52.983253 10967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:52.986138 11085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:00:52.991832 11085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Bootstrap starting.
I20260812 07:00:52.992293 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 10967
I20260812 07:00:52.992448 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:00:52.992767 28091 external_mini_cluster.cc:1428] Setting key c6abdec7ad772d6e4d30d02ec0020e4d
I20260812 07:00:52.994755 11085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Neither blocks nor log segments found. Creating new log.
I20260812 07:00:52.995581 11085 log.cc:824] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:52.997474 11085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: No bootstrap required, opened a new log
I20260812 07:00:53.001156 11085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:53.001380 11085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:00:53.001456 11085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80152bdbc7f240e6871e1619ab4bb9ad, State: Initialized, Role: FOLLOWER
Aug 12 07:00:52 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:53.001966 11085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:53.002099 11085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:53.002185 11085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:53.002305 11085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:00:53.003523 11085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:53.003891 11085 leader_election.cc:302] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80152bdbc7f240e6871e1619ab4bb9ad; no voters:
I20260812 07:00:53.004204 11085 leader_election.cc:288] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:00:53.004348 11096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:00:53.004563 11096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 LEADER]: Becoming Leader. State: Replica: 80152bdbc7f240e6871e1619ab4bb9ad, State: Running, Role: LEADER
I20260812 07:00:53.004984 11096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:53.005478 11085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:53.008042 11097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } } }
I20260812 07:00:53.008318 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:52.994223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.27.110.254:36415)
0812 07:00:52.994664 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:52.994684 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:52.994726 (+ 42us) server_negotiation.cc:407] Connection header received
0812 07:00:52.995436 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:52.995455 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:52.995849 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:52.996198 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:52.997012 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:52.998046 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:52.998922 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:52.999475 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:52.999664 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.002431 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:53.002437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:53.002454 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:53.002487 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:53.004548 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.005095 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.005101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.005106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.005258 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.005627 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.005630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.005632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.006148 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:53.006358 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:53.006851 (+ 493us) server_negotiation.cc:299] Negotiation successful
0812 07:00:53.007205 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260812 07:00:53.008339 11097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:53.008257 11098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80152bdbc7f240e6871e1619ab4bb9ad. Latest consensus state: current_term: 1 leader_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } } }
I20260812 07:00:53.008824 11098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: This master's current role is: LEADER
W20260812 07:00:53.010861 11110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:00:53.010928 11110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:00:53.011010 11114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:53.011730 11114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:53.016916 11114 catalog_manager.cc:1347] Generated new cluster ID: 557bd00e828e481a9406fb51602bdb8c
I20260812 07:00:53.016986 11114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:53.032532 11114 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:00:53.033644 11114 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:53.039484 11114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Generated new TSK 0
I20260812 07:00:53.040213 11114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:00:53.077883 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36415
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:53.198002 11122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:53.198256 11122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:53.198328 11122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:53.202356 11122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:53.202414 11122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:53.202502 11122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:53.208091 11122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:53.209141 11122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:53.210173 11122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:00:53.219904 11122 server_base.cc:1034] running on GCE node
W20260812 07:00:53.219833 11130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:53.219832 11131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:53.219842 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:53.220544 11122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:53.221204 11122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:53.222371 11122 hybrid_clock.cc:648] HybridClock initialized: now 1786518053222344 us; error 38 us; skew 500 ppm
Aug 12 07:00:53 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:53.225247 11122 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:00:53.226140 11122 webserver.cc:460] Webserver started at http://127.27.110.193:33323/ using document root <none> and password file <none>
I20260812 07:00:53.226823 11122 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:53.226918 11122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:53.227188 11122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:53.229208 11122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "5af3e627c7414135a3da57b573a6577f"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "a56c1f22cb0fae93d40ac27b78c2f4c0"
server_key_iv: "a3b6060c7c6fb2c05a20a9d4a329a724"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.229732 11122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "5af3e627c7414135a3da57b573a6577f"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "a56c1f22cb0fae93d40ac27b78c2f4c0"
server_key_iv: "a3b6060c7c6fb2c05a20a9d4a329a724"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.233649 11122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:00:53.236418 11150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.237701 11122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:53.237800 11122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "5af3e627c7414135a3da57b573a6577f"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "a56c1f22cb0fae93d40ac27b78c2f4c0"
server_key_iv: "a3b6060c7c6fb2c05a20a9d4a329a724"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.237892 11122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:53.257079 11122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:53.260035 11122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:53.260206 11122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:53.260751 11122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:53.261551 11122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:53.261600 11122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.261641 11122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:53.261657 11122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.271862 11122 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:40225
I20260812 07:00:53.271889 11349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:40225 every 8 connection(s)
I20260812 07:00:53.272778 11122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:53.276090 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 11122
I20260812 07:00:53.276224 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:00:53.276525 28091 external_mini_cluster.cc:1428] Setting key 8f463508e12584b9fe20e85152e8deea
Aug 12 07:00:53 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:00:53.288546 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:53.274578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:38647 (local address 127.27.110.254:36415)
0812 07:00:53.274742 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:53.274748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:53.275732 (+ 984us) server_negotiation.cc:407] Connection header received
0812 07:00:53.277189 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:53.277193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:53.277246 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:53.277346 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:53.278819 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.279696 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.281350 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.281588 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.281677 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.284704 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:53.284707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:53.284710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:53.284735 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:53.286245 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.286793 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.286801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.286803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.286888 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.287325 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.287329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.287330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.287525 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:53.287636 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:53.288228 (+ 592us) server_negotiation.cc:299] Negotiation successful
0812 07:00:53.288364 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:00:53.289371 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:53.275002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:38647)
0812 07:00:53.275568 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:53.275616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:53.276943 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:53.277595 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:53.277604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:53.278170 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:53.278666 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:53.278679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.279838 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.279840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:53.281211 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:53.281226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.281754 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.281761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:53.281995 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.282648 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:53.282672 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:53.284512 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:53.286380 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:53.286386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:53.286401 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:53.286672 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:53.286986 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:53.286989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:53.286991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:53.287225 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:53.287647 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:53.287656 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:53.287914 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 07:00:53.288146 (+ 232us) client_negotiation.cc:240] Negotiation successful
0812 07:00:53.288494 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:00:53.290694 11351 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36415
I20260812 07:00:53.290953 11351 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:53.291544 11351 heartbeater.cc:511] Master 127.27.110.254:36415 requested a full tablet report, sending...
I20260812 07:00:53.293308 11024 ts_manager.cc:194] Registered new tserver with Master: 5af3e627c7414135a3da57b573a6577f (127.27.110.193:40225)
I20260812 07:00:53.295017 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:38647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:00:53.306311 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36415
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:53.418776 11361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:53.419009 11361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:53.419050 11361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:53.423007 11361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:53.423064 11361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:53.423195 11361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:53.428296 11361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:53.429268 11361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:53.430207 11361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:53.440049 11370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:53.440047 11369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:53.440058 11373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:53.440557 11361 server_base.cc:1034] running on GCE node
I20260812 07:00:53.440980 11361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:53.441574 11361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:53.442744 11361 hybrid_clock.cc:648] HybridClock initialized: now 1786518053442703 us; error 59 us; skew 500 ppm
Aug 12 07:00:53 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:53.445716 11361 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:00:53.446616 11361 webserver.cc:460] Webserver started at http://127.27.110.194:46829/ using document root <none> and password file <none>
I20260812 07:00:53.447281 11361 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:53.447348 11361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:53.447589 11361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:53.449613 11361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "5f4f280b5bbb43b89e55de82819e11c1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "f424d2a1532f84ef44cfb1336ebf72f4"
server_key_iv: "b3795f62278e07816c3f4f2c16d2566c"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.450188 11361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "5f4f280b5bbb43b89e55de82819e11c1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "f424d2a1532f84ef44cfb1336ebf72f4"
server_key_iv: "b3795f62278e07816c3f4f2c16d2566c"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.454200 11361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 07:00:53.456660 11381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.457865 11361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:00:53.457991 11361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "5f4f280b5bbb43b89e55de82819e11c1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "f424d2a1532f84ef44cfb1336ebf72f4"
server_key_iv: "b3795f62278e07816c3f4f2c16d2566c"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.458093 11361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:53.491503 11361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:53.494995 11361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:53.495229 11361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:53.495852 11361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:53.496776 11361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:53.496825 11361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.496889 11361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:53.496923 11361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.506379 11361 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:46871
I20260812 07:00:53.506409 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:46871 every 8 connection(s)
I20260812 07:00:53.507504 11361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:53.513492 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 11361
I20260812 07:00:53.513624 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:00:53.513934 28091 external_mini_cluster.cc:1428] Setting key de0ef88b7905aec56ee59b19449558de
Aug 12 07:00:53 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:53.521966 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:53.509591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:58465 (local address 127.27.110.254:36415)
0812 07:00:53.509747 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:53.509752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:53.510929 (+ 1177us) server_negotiation.cc:407] Connection header received
0812 07:00:53.511891 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:53.511895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:53.511952 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:53.512030 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:53.513237 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.513837 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.515219 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.515454 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.515579 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.518221 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:53.518225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:53.518227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:53.518249 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:53.519799 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.520340 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.520351 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.520353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.520433 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.520789 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.520794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.520796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.520979 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:53.521093 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:53.521735 (+ 642us) server_negotiation.cc:299] Negotiation successful
0812 07:00:53.521847 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:00:53.523303 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:53.510053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:58465)
0812 07:00:53.510772 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:53.510817 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:53.511673 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:53.512202 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:53.512209 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:53.512607 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:53.513096 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:53.513109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.513972 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.513977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:53.514999 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:53.515010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.515619 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.515623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:53.515796 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.516412 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:53.516428 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:53.518046 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:53.519929 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:53.519935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:53.519950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:53.520221 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:53.520532 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:53.520535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:53.520537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:53.520688 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:53.521093 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:53.521101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:53.521434 (+ 333us) client_negotiation.cc:769] Sending connection context
0812 07:00:53.521690 (+ 256us) client_negotiation.cc:240] Negotiation successful
0812 07:00:53.522079 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":112,"threads_started":1}
I20260812 07:00:53.524602 11577 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36415
I20260812 07:00:53.524849 11577 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:00:53.525349 11577 heartbeater.cc:511] Master 127.27.110.254:36415 requested a full tablet report, sending...
I20260812 07:00:53.526713 11024 ts_manager.cc:194] Registered new tserver with Master: 5f4f280b5bbb43b89e55de82819e11c1 (127.27.110.194:46871)
I20260812 07:00:53.527408 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:58465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:00:53.544322 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36415
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:53.660120 11583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:53.660373 11583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:53.660413 11583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:53.664458 11583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:53.664517 11583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:53.664579 11583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:53.669723 11583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:53.670789 11583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:53.671873 11583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:53.681505 11595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:53.681512 11592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:53.681653 11583 server_base.cc:1034] running on GCE node
W20260812 07:00:53.681516 11593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:53.682219 11583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:53.682873 11583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:53.684033 11583 hybrid_clock.cc:648] HybridClock initialized: now 1786518053684011 us; error 36 us; skew 500 ppm
Aug 12 07:00:53 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:53.686841 11583 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:00:53.687811 11583 webserver.cc:460] Webserver started at http://127.27.110.195:33547/ using document root <none> and password file <none>
I20260812 07:00:53.688374 11583 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:53.688437 11583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:53.688740 11583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:00:53.690842 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "d9dd2857f0574ef9901e495bf3373ab1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "4cca2a0b56a9bb452f7c48fa4b515e6e"
server_key_iv: "560fab974097eee28349755c61bf40cc"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.691475 11583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "d9dd2857f0574ef9901e495bf3373ab1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "4cca2a0b56a9bb452f7c48fa4b515e6e"
server_key_iv: "560fab974097eee28349755c61bf40cc"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.695258 11583 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:00:53.697710 11604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.698920 11583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:53.699043 11583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "d9dd2857f0574ef9901e495bf3373ab1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "4cca2a0b56a9bb452f7c48fa4b515e6e"
server_key_iv: "560fab974097eee28349755c61bf40cc"
server_key_version: "encryptionkey@0"
I20260812 07:00:53.699203 11583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:53.726662 11583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:53.730018 11583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:53.730244 11583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:53.730895 11583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:53.731837 11583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:53.731887 11583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.731961 11583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:53.732002 11583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:53.741348 11583 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44765
I20260812 07:00:53.741443 11811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44765 every 8 connection(s)
I20260812 07:00:53.742424 11583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:00:53.751436 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 11583
I20260812 07:00:53.751593 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
Aug 12 07:00:53 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518053, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:00:53.751932 28091 external_mini_cluster.cc:1428] Setting key 66e000217c83916f055662d0617b7444
I20260812 07:00:53.756055 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:53.744377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:55919 (local address 127.27.110.254:36415)
0812 07:00:53.744523 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:53.744527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:53.745599 (+ 1072us) server_negotiation.cc:407] Connection header received
0812 07:00:53.746402 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:53.746405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:53.746451 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:53.746524 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:53.747750 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.748267 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.749363 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.749582 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.749662 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.752367 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:53.752370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:53.752372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:53.752398 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:53.754193 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.754699 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.754702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.754703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.754779 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:53.755127 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:53.755130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:53.755132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:53.755313 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:53.755393 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:53.755844 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 07:00:53.755939 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:00:53.757097 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:53.744761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:55919)
0812 07:00:53.745426 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:53.745468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:53.746224 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:53.746708 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:53.746715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:53.747058 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:53.747611 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:53.747624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.748375 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.748378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:53.749251 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:53.749261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:53.749728 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:53.749731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:53.749872 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:53.750410 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:53.750426 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:53.752180 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:00:53.754323 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:53.754328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:53.754343 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:53.754609 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:53.754862 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:00:53.754864 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:00:53.754866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:00:53.754996 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:00:53.755409 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:00:53.755414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:00:53.755633 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 07:00:53.755794 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 07:00:53.756032 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":118,"threads_started":1}
I20260812 07:00:53.758286 11813 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36415
I20260812 07:00:53.758607 11813 heartbeater.cc:464] Registering TS with master...
I20260812 07:00:53.759174 11813 heartbeater.cc:511] Master 127.27.110.254:36415 requested a full tablet report, sending...
I20260812 07:00:53.760197 11024 ts_manager.cc:194] Registered new tserver with Master: d9dd2857f0574ef9901e495bf3373ab1 (127.27.110.195:44765)
I20260812 07:00:53.760788 11024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:55919
I20260812 07:00:53.765919 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab.
Entry for principal oryx/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab.
WARNING: no policy specified for oryx/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab.
Entry for principal oryx/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab.
WARNING: no policy specified for oryx/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab.
Entry for principal oryx/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab.
I20260812 07:00:53.839955 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 11122
I20260812 07:00:53.852710 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 11361
I20260812 07:00:53.882462 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 11583
I20260812 07:00:53.896602 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 10967
I20260812 07:00:53.919464 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36415
--webserver_interface=127.27.110.254
--webserver_port=45873
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36415
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:00:54.032226 11822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:54.032509 11822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:54.032552 11822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:54.036648 11822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:00:54.036702 11822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:54.036722 11822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:54.036739 11822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:00:54.036754 11822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:00:54.041957 11822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=45873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:54.042985 11822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:54.044077 11822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:54.052542 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:54.052549 11835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:54.052573 11833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:54.052760 11822 server_base.cc:1034] running on GCE node
I20260812 07:00:54.053398 11822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:54.054438 11822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:54.055608 11822 hybrid_clock.cc:648] HybridClock initialized: now 1786518054055591 us; error 29 us; skew 500 ppm
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518054, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:54.058243 11822 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:00:54.059408 11822 webserver.cc:460] Webserver started at http://127.27.110.254:45873/ using document root <none> and password file <none>
I20260812 07:00:54.060035 11822 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:54.060133 11822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:54.063807 11822 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:54.065907 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.067236 11822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:00:54.067369 11822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "80152bdbc7f240e6871e1619ab4bb9ad"
format_stamp: "Formatted at 2026-08-12 07:00:52 on dist-test-slave-w4v5"
server_key: "ec81f4ed875d0744671afa04ea282467"
server_key_iv: "30ad260aabba82ddd7ab7ae0a6fa1205"
server_key_version: "encryptionkey@0"
I20260812 07:00:54.067912 11822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:54.078080 11822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:54.080956 11822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:54.081156 11822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:54.088433 11822 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36415
I20260812 07:00:54.088459 11926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36415 every 8 connection(s)
I20260812 07:00:54.089527 11822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:00:54.093736 11927 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:00:54.095449 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Bootstrap starting.
I20260812 07:00:54.096691 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 11822
I20260812 07:00:54.097240 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:40225
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:36415
--webserver_port=33323
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:00:54.114324 11927 log.cc:824] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Log is configured to *not* fsync() on all Append() calls
I20260812 07:00:54.121336 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:00:54.121757 11927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Bootstrap complete.
I20260812 07:00:54.126255 11927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:54.126653 11927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80152bdbc7f240e6871e1619ab4bb9ad, State: Initialized, Role: FOLLOWER
I20260812 07:00:54.127246 11927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:54.127372 11927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:00:54.127460 11927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:00:54.127559 11927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:00:54.129552 11927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:54.129892 11927 leader_election.cc:302] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80152bdbc7f240e6871e1619ab4bb9ad; no voters:
I20260812 07:00:54.130241 11927 leader_election.cc:288] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:00:54.130367 11934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:00:54.130605 11934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [term 2 LEADER]: Becoming Leader. State: Replica: 80152bdbc7f240e6871e1619ab4bb9ad, State: Running, Role: LEADER
I20260812 07:00:54.130908 11934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } }
I20260812 07:00:54.131538 11927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:00:54.132340 11936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } } }
I20260812 07:00:54.132475 11936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:54.132791 11937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80152bdbc7f240e6871e1619ab4bb9ad. Latest consensus state: current_term: 2 leader_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80152bdbc7f240e6871e1619ab4bb9ad" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36415 } } }
I20260812 07:00:54.132937 11937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad [sys.catalog]: This master's current role is: LEADER
I20260812 07:00:54.136749 11963 catalog_manager.cc:1259] Loaded cluster ID: 557bd00e828e481a9406fb51602bdb8c
I20260812 07:00:54.136812 11963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: loading cluster ID for follower catalog manager: success
I20260812 07:00:54.139184 11963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: acquiring CA information for follower catalog manager: success
I20260812 07:00:54.140553 11963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:00:54.140676 11952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:00:54.141566 11952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:00:54.142005 11952 catalog_manager.cc:1259] Loaded cluster ID: 557bd00e828e481a9406fb51602bdb8c
I20260812 07:00:54.142079 11952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:00:54.142704 11952 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:00:54.143196 11952 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 80152bdbc7f240e6871e1619ab4bb9ad: Loaded TSK: 0
I20260812 07:00:54.143903 11952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:00:54.218557 11932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:54.218812 11932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:54.218854 11932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:54.222874 11932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:54.222940 11932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:54.222999 11932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:00:54.228364 11932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.193.keytab
--principal=oryx/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:40225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=33323
--tserver_master_addrs=127.27.110.254:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:54.229455 11932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:54.230551 11932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:54.241120 11976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:54.241506 11932 server_base.cc:1034] running on GCE node
W20260812 07:00:54.241586 11978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:54.241178 11975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:54.242041 11932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:54.242798 11932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:54.244007 11932 hybrid_clock.cc:648] HybridClock initialized: now 1786518054243991 us; error 35 us; skew 500 ppm
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518054, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:54.247032 11932 init.cc:377] Logged in from keytab as oryx/127.27.110.193@KRBTEST.COM (short username oryx)
I20260812 07:00:54.248088 11932 webserver.cc:460] Webserver started at http://127.27.110.193:33323/ using document root <none> and password file <none>
I20260812 07:00:54.248780 11932 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:54.248889 11932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:54.252971 11932 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:00:54.255227 11994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.256534 11932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:00:54.256641 11932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "5af3e627c7414135a3da57b573a6577f"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "a56c1f22cb0fae93d40ac27b78c2f4c0"
server_key_iv: "a3b6060c7c6fb2c05a20a9d4a329a724"
server_key_version: "encryptionkey@0"
I20260812 07:00:54.257200 11932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:54.277014 11932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:54.280041 11932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:54.280263 11932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:54.280923 11932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:54.281816 11932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:54.281864 11932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.281937 11932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:54.281968 11932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.291808 11932 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:40225
I20260812 07:00:54.291842 12189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:40225 every 8 connection(s)
I20260812 07:00:54.292851 11932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:00:54.295264 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 11932
I20260812 07:00:54.295866 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:46871
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:36415
--webserver_port=46829
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:00:54.308777 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.295032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:42815 (local address 127.27.110.254:36415)
0812 07:00:54.295750 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.295771 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.296226 (+ 455us) server_negotiation.cc:407] Connection header received
0812 07:00:54.297868 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.297888 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.298323 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.298703 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.300081 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.301079 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.303042 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.303580 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.303705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.308076 (+ 4371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:42815: BlockingRecv error: recv got EOF from 127.27.110.193:42815 (error 108)
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":141,"threads_started":1}
I20260812 07:00:54.308959 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.295083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:42815)
0812 07:00:54.296012 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.296078 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.297353 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.298765 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.298772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.299207 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.299884 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.299909 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.301262 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.301269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.302895 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.302911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.303772 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.303779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.304030 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.304699 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.304726 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.307886 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":130,"threads_started":1}
W20260812 07:00:54.309547 12190 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:36415 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.110.254:36415: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.315454 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.310060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:58501)
0812 07:00:54.310181 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.310196 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.310269 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.310528 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.310530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.310656 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.310766 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.310770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.311829 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.311832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.312369 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.312376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.312951 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.312954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.313021 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.313518 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.313530 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.315363 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 07:00:54.315651 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.310155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:58501 (local address 127.27.110.254:36415)
0812 07:00:54.310273 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.310277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.310328 (+ 51us) server_negotiation.cc:407] Connection header received
0812 07:00:54.310366 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.310369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.310414 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.310507 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.310872 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.311726 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.312483 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.312842 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.312943 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.315526 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:58501: BlockingRecv error: recv got EOF from 127.27.110.193:58501 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.321235 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.316125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:39387)
0812 07:00:54.316219 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.316234 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.316309 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.316531 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.316533 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.316652 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.316734 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.316738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.317727 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.317729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.318231 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.318238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.318805 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.318807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.318863 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.319404 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.319415 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.321146 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260812 07:00:54.321419 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.316204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:39387 (local address 127.27.110.254:36415)
0812 07:00:54.316315 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.316319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.316358 (+ 39us) server_negotiation.cc:407] Connection header received
0812 07:00:54.316392 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.316395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.316436 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.316508 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.316836 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.317617 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.318346 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.318696 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.318796 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.321314 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:39387: BlockingRecv error: recv got EOF from 127.27.110.193:39387 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 07:00:54.321687 12190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:00:54.416060 12198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:54.416332 12198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:54.416378 12198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:54.420421 12198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:54.420483 12198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:54.420591 12198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:00:54.425833 12198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.194.keytab
--principal=oryx/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:46871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=46829
--tserver_master_addrs=127.27.110.254:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:54.426900 12198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:54.428037 12198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:54.437712 12207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:54.437723 12211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:54.437866 12198 server_base.cc:1034] running on GCE node
W20260812 07:00:54.438031 12206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:54.438484 12198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:54.439180 12198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:54.440351 12198 hybrid_clock.cc:648] HybridClock initialized: now 1786518054440331 us; error 34 us; skew 500 ppm
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518054, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:54.443248 12198 init.cc:377] Logged in from keytab as oryx/127.27.110.194@KRBTEST.COM (short username oryx)
I20260812 07:00:54.444123 12198 webserver.cc:460] Webserver started at http://127.27.110.194:46829/ using document root <none> and password file <none>
I20260812 07:00:54.444631 12198 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:54.444701 12198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:54.448582 12198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:00:54.450601 12221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.451936 12198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:00:54.452057 12198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "5f4f280b5bbb43b89e55de82819e11c1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "f424d2a1532f84ef44cfb1336ebf72f4"
server_key_iv: "b3795f62278e07816c3f4f2c16d2566c"
server_key_version: "encryptionkey@0"
I20260812 07:00:54.452589 12198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:54.465848 12198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:54.468626 12198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:54.468775 12198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:54.469295 12198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:54.470057 12198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:54.470103 12198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.470141 12198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:54.470157 12198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.479494 12198 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:46871
I20260812 07:00:54.479516 12400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:46871 every 8 connection(s)
I20260812 07:00:54.480476 12198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:00:54.483899 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 12198
I20260812 07:00:54.484354 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:44765
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:36415
--webserver_port=33547
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:00:54.492116 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.482307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:42659 (local address 127.27.110.254:36415)
0812 07:00:54.482451 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.482455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.483563 (+ 1108us) server_negotiation.cc:407] Connection header received
0812 07:00:54.484578 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.484589 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.484645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.484703 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.486032 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.486574 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.487848 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.488079 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.488206 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.491947 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:42659: BlockingRecv error: recv got EOF from 127.27.110.194:42659 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:00:54.492573 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.482703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:42659)
0812 07:00:54.483411 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.483454 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.484314 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.485039 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.485045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.485396 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.485875 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.485888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.486725 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.486729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.487725 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.487736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.488243 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.488247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.488419 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.489036 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.489052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.491744 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":111,"threads_started":1}
W20260812 07:00:54.493191 12401 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:36415 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.110.254:36415: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.499367 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.493699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:54611)
0812 07:00:54.493809 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.493825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.493904 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.494194 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.494196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.494327 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.494424 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.494428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.495589 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.495592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.496130 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.496138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.496808 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.496811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.496878 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.497393 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.497405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.499269 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 07:00:54.499755 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.493786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:54611 (local address 127.27.110.254:36415)
0812 07:00:54.493932 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.493939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.493958 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:00:54.494009 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.494015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.494071 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.494174 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.494553 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.495457 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.496260 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.496691 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.496824 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.499422 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:54611: BlockingRecv error: recv got EOF from 127.27.110.194:54611 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.504829 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.499911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:48697)
0812 07:00:54.500136 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.500148 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.500239 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.500495 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.500498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.500642 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.500722 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.500727 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.501449 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.501457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.501962 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.501970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.502412 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.502415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.502493 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.502970 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.502982 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.504744 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170}
W20260812 07:00:54.505040 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.500147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:48697 (local address 127.27.110.254:36415)
0812 07:00:54.500299 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.500305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.500319 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:54.500358 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.500360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.500393 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.500479 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.500835 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.501327 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.502077 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.502300 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.502409 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.504893 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:48697: BlockingRecv error: recv got EOF from 127.27.110.194:48697 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 07:00:54.505096 12401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:00:54.602947 12406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:00:54.603268 12406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:00:54.603348 12406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:00:54.607293 12406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:00:54.607344 12406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:00:54.607462 12406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:00:54.612720 12406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/krb5kdc/oryx_127.27.110.195.keytab
--principal=oryx/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:44765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=33547
--tserver_master_addrs=127.27.110.254:36415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:00:54.613741 12406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:00:54.614774 12406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:00:54.624495 12416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:54.624528 12414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:00:54.624527 12419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:00:54.624758 12406 server_base.cc:1034] running on GCE node
I20260812 07:00:54.625150 12406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:00:54.625793 12406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:00:54.626987 12406 hybrid_clock.cc:648] HybridClock initialized: now 1786518054626970 us; error 28 us; skew 500 ppm
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518054, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:00:54.629819 12406 init.cc:377] Logged in from keytab as oryx/127.27.110.195@KRBTEST.COM (short username oryx)
I20260812 07:00:54.630717 12406 webserver.cc:460] Webserver started at http://127.27.110.195:33547/ using document root <none> and password file <none>
I20260812 07:00:54.631349 12406 fs_manager.cc:359] Metadata directory not provided
I20260812 07:00:54.631466 12406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:00:54.635828 12406 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:00:54.637981 12432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.639199 12406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:00:54.639338 12406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "d9dd2857f0574ef9901e495bf3373ab1"
format_stamp: "Formatted at 2026-08-12 07:00:53 on dist-test-slave-w4v5"
server_key: "4cca2a0b56a9bb452f7c48fa4b515e6e"
server_key_iv: "560fab974097eee28349755c61bf40cc"
server_key_version: "encryptionkey@0"
I20260812 07:00:54.639869 12406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:00:54.657100 12406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:00:54.660410 12406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:00:54.660650 12406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:00:54.661237 12406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:00:54.662091 12406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:00:54.662166 12406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.662237 12406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:00:54.662288 12406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:00:54.671465 12406 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44765
I20260812 07:00:54.671556 12619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44765 every 8 connection(s)
I20260812 07:00:54.672523 12406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.682078 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 12406
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 07:00:54.683475 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.674241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:55251 (local address 127.27.110.254:36415)
0812 07:00:54.674376 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.674380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.675473 (+ 1093us) server_negotiation.cc:407] Connection header received
0812 07:00:54.676384 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.676387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.676443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.676554 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.677731 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.678254 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.679495 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.679725 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.679833 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.683347 (+ 3514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:55251: BlockingRecv error: recv got EOF from 127.27.110.195:55251 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:00:54.684127 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.674628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:55251)
0812 07:00:54.675332 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.675373 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.676179 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.676750 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.676758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.677097 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.677583 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.677597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.678409 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.678412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.679367 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.679377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.679887 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.679892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.680051 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.680639 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.680656 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.683218 (+ 2562us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":113,"threads_started":1}
W20260812 07:00:54.684682 12620 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:36415 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.110.254:36415: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.690600 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.685084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:36949)
0812 07:00:54.685258 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.685271 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.685345 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.685589 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.685591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.685714 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.685801 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.685806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.686676 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.686679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.687335 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.687345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.687962 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.687964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.688030 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.688453 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.688469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.690467 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260812 07:00:54.690752 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.685201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:36949 (local address 127.27.110.254:36415)
0812 07:00:54.685351 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.685355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.685389 (+ 34us) server_negotiation.cc:407] Connection header received
0812 07:00:54.685424 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.685427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.685468 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.685543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.685911 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.686550 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.687460 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.687851 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.687971 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.690602 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:36949: BlockingRecv error: recv got EOF from 127.27.110.195:36949 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 07:00:54.692761 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.683205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.27.110.254:36415)
0812 07:00:54.683444 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.683448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.683462 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:54.683499 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.683502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.683550 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.683716 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.684499 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.685094 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.686250 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.686479 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.686555 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.688212 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:00:54.688221 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:00:54.688239 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:00:54.688297 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:00:54.690260 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:54.690919 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:54.690925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:54.690931 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:54.691118 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:00:54.691464 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:00:54.691468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:00:54.691470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:00:54.691828 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:00:54.692042 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 07:00:54.692353 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 07:00:54.692595 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:54 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:54.696146 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:54.691224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:33935)
0812 07:00:54.691335 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:54.691349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:54.691420 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:54.691641 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:54.691644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:54.691794 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:54.691857 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.691862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.692573 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.692577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.693200 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:54.693210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.693743 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.693746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:54.693823 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.694212 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:54.694226 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:54.696062 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260812 07:00:54.696278 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:54.691270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:33935 (local address 127.27.110.254:36415)
0812 07:00:54.691399 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:54.691403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:54.691415 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:00:54.691497 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:54.691500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:54.691535 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:54.691627 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:54.691970 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.692468 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.693311 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:54.693606 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:54.693718 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:54.696177 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:33935: BlockingRecv error: recv got EOF from 127.27.110.195:33935 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 07:00:54.696393 12620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 07:00:55 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:55 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:55.332839 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:55.322312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:54091)
0812 07:00:55.322671 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:55.322686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:55.322780 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:55.323013 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:55.323015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:55.323279 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:55.323409 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:55.323416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.324189 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.324192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:55.324807 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:55.324816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.325306 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.325309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:55.325393 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:55.330410 (+ 5017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:55.330424 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:55.332714 (+ 2290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
W20260812 07:00:55.332998 12636 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:55.322453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:54091 (local address 127.27.110.254:36415)
0812 07:00:55.322718 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:55.322721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:55.322736 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:00:55.322872 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:55.322874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:55.322923 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:55.323001 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:55.323521 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.324078 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.324922 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.325172 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.325255 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:55.332854 (+ 7599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:54091: BlockingRecv error: recv got EOF from 127.27.110.193:54091 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
Aug 12 07:00:55 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:55 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:55.511389 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:55.505657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:53809)
0812 07:00:55.505994 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:55.506007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:55.506151 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:55.506428 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:55.506431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:55.506649 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:55.506765 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:55.506771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.507625 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.507630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:55.508214 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:55.508222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.508703 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.508706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:55.508806 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:55.509372 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:55.509384 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:55.511218 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":106,"threads_started":1}
W20260812 07:00:55.511456 12636 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:55.505770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:53809 (local address 127.27.110.254:36415)
0812 07:00:55.505948 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:55.505953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:55.506083 (+ 130us) server_negotiation.cc:407] Connection header received
0812 07:00:55.506254 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:55.506259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:55.506315 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:55.506404 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:55.506890 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.507503 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.508338 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.508581 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.508699 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:55.511345 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:53809: BlockingRecv error: recv got EOF from 127.27.110.194:53809 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 07:00:55 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:55 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:55.703032 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:55.697131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:49733)
0812 07:00:55.697417 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:55.697434 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:55.697561 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:55.697913 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:55.697917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:55.698176 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:55.698311 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:55.698322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.699169 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.699173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:55.699752 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:55.699761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.700205 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.700207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:55.700268 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:55.700840 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:55.700851 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:55.702895 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
W20260812 07:00:55.703279 12636 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:55.697179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:49733 (local address 127.27.110.254:36415)
0812 07:00:55.697332 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:55.697336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:55.697534 (+ 198us) server_negotiation.cc:407] Connection header received
0812 07:00:55.697692 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:55.697696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:55.697760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:55.697854 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:55.698454 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.699051 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.699876 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:55.700099 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:55.700185 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:55.703034 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:49733: BlockingRecv error: recv got EOF from 127.27.110.195:49733 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 07:00:56 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:56 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:56.339649 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:56.333789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:38495)
0812 07:00:56.334130 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:56.334146 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:56.334297 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:56.334583 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:56.334586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:56.334816 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:56.334914 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:56.334920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.335875 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.335880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:56.336446 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:56.336454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.336933 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.336936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:56.337023 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:56.337507 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:56.337519 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:56.339447 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":113,"threads_started":1}
W20260812 07:00:56.339689 12642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:56.333853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:38495 (local address 127.27.110.254:36415)
0812 07:00:56.334124 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:56.334129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:56.334234 (+ 105us) server_negotiation.cc:407] Connection header received
0812 07:00:56.334402 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:56.334405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:56.334462 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:56.334560 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:56.335038 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.335761 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.336570 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.336817 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.336927 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:56.339567 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:38495: BlockingRecv error: recv got EOF from 127.27.110.193:38495 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":66,"threads_started":1}
Aug 12 07:00:56 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:56 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:56.518153 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:56.512294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:38253)
0812 07:00:56.512572 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:56.512587 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:56.512693 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:56.512958 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:56.512960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:56.513148 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:56.513259 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:56.513264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.514254 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.514261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:56.514919 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:56.514929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.515460 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.515463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:56.515552 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:56.516125 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:56.516138 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:56.518020 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":71,"threads_started":1}
W20260812 07:00:56.518397 12642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:56.512430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:38253 (local address 127.27.110.254:36415)
0812 07:00:56.512617 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:56.512621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:56.512642 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:00:56.512784 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:56.512786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:56.512831 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:56.512911 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:56.513391 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.514131 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.515052 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.515334 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.515444 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:56.518159 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:38253: BlockingRecv error: recv got EOF from 127.27.110.194:38253 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 07:00:56 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:56 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:56.709388 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:56.703857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:44483)
0812 07:00:56.704214 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:56.704230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:56.704330 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:56.704595 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:56.704597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:56.704787 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:56.704888 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:56.704892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.705697 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.705700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:56.706272 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:56.706281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.706768 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.706771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:56.706839 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:56.707371 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:56.707383 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:56.709247 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":165,"threads_started":1}
W20260812 07:00:56.709542 12642 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:56.704124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:44483 (local address 127.27.110.254:36415)
0812 07:00:56.704260 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:56.704263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:56.704281 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:00:56.704417 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:56.704420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:56.704466 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:56.704547 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:56.705010 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.705560 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.706408 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:56.706640 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:56.706732 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:56.709385 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:44483: BlockingRecv error: recv got EOF from 127.27.110.195:44483 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 07:00:57 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:57 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:57.346907 12647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:57.340614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:56855)
0812 07:00:57.340935 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:57.340950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:57.341061 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:57.341374 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:57.341376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:57.341569 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:57.341674 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:57.341679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.342558 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.342561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:57.343325 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:57.343340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.343985 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.343989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:57.344091 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:57.344765 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:57.344780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:57.346772 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":120,"threads_started":1}
W20260812 07:00:57.347059 12648 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:57.340735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:56855 (local address 127.27.110.254:36415)
0812 07:00:57.341072 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:57.341077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:57.341098 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:00:57.341171 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:57.341174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:57.341234 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:57.341329 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:57.341852 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.342419 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.343467 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.343803 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.343933 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:57.346914 (+ 2981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:56855: BlockingRecv error: recv got EOF from 127.27.110.193:56855 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":109,"threads_started":1}
Aug 12 07:00:57 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:57 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:57.524449 12649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:57.518913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:47663)
0812 07:00:57.519282 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:57.519295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:57.519396 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:57.519637 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:57.519640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:57.519831 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:57.519929 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:57.519934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.520659 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.520662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:57.521230 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:57.521238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.521687 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.521689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:57.521753 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:57.522340 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:57.522352 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:57.524324 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
W20260812 07:00:57.524619 12648 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:57.519047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:47663 (local address 127.27.110.254:36415)
0812 07:00:57.519235 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:57.519239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:57.519366 (+ 127us) server_negotiation.cc:407] Connection header received
0812 07:00:57.519475 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:57.519477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:57.519521 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:57.519601 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:57.520028 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.520551 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.521339 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.521558 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.521655 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:57.524460 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:47663: BlockingRecv error: recv got EOF from 127.27.110.194:47663 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 07:00:57 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:57 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:57.722033 12652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:57.710351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:55741)
0812 07:00:57.710719 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:57.710735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:57.710866 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:57.711237 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:57.711240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:57.711519 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:57.711671 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:57.711683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.712671 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.712673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:57.713250 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:57.713258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.713715 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.713717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:57.713783 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:57.719836 (+ 6053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:57.719852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:57.721901 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
W20260812 07:00:57.722286 12648 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:57.710475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:55741 (local address 127.27.110.254:36415)
0812 07:00:57.710651 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:57.710656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:57.710844 (+ 188us) server_negotiation.cc:407] Connection header received
0812 07:00:57.710970 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:57.710985 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:57.711052 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:57.711179 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:57.711824 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.712553 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.713370 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:57.713609 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:57.713695 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:57.722105 (+ 8410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:55741: BlockingRecv error: recv got EOF from 127.27.110.195:55741 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 07:00:58 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:58 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:58.353585 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:58.347703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:59219)
0812 07:00:58.348011 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:58.348028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:58.348170 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:58.348448 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:58.348451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:58.348706 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:58.348857 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:58.348870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.349783 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.349786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:58.350344 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:58.350353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.350856 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.350859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:58.350925 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:58.351442 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:58.351454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:58.353454 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":72,"threads_started":1}
W20260812 07:00:58.353863 12656 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:58.347856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:59219 (local address 127.27.110.254:36415)
0812 07:00:58.348216 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:58.348219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:58.348233 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:00:58.348268 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:58.348270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:58.348311 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:58.348387 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:58.348980 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.349626 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.350503 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.350742 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.350834 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:58.353622 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:59219: BlockingRecv error: recv got EOF from 127.27.110.193:59219 (error 108)
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
Aug 12 07:00:58 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:58 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:58.531040 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:58.525261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:38151)
0812 07:00:58.525538 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:58.525552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:58.525659 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:58.525941 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:58.525944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:58.526153 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:58.526261 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:58.526267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.527007 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.527011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:58.527672 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:58.527682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.528127 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.528130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:58.528200 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:58.528851 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:58.528865 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:58.530913 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
W20260812 07:00:58.531337 12656 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:58.525351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:38151 (local address 127.27.110.254:36415)
0812 07:00:58.525499 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:58.525503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:58.525609 (+ 106us) server_negotiation.cc:407] Connection header received
0812 07:00:58.525739 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:58.525741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:58.525785 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:58.525848 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:58.526376 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.526897 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.527788 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.528009 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.528091 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:58.531175 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:38151: BlockingRecv error: recv got EOF from 127.27.110.194:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 07:00:58 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:58 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:58.729537 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:58.723035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:33901)
0812 07:00:58.723431 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:58.723450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:58.723666 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:58.723996 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:58.724000 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:58.724249 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:58.724357 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:58.724363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.725227 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.725232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:58.725869 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:58.725878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.726403 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.726407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:58.726508 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:58.727124 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:58.727138 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:58.729354 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":162,"threads_started":1}
W20260812 07:00:58.729640 12656 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:58.723222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:33901 (local address 127.27.110.254:36415)
0812 07:00:58.723408 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:58.723414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:58.723593 (+ 179us) server_negotiation.cc:407] Connection header received
0812 07:00:58.723783 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:58.723787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:58.723848 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:58.723952 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:58.724498 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.725096 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.726007 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:58.726269 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:58.726396 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:58.729481 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:33901: BlockingRecv error: recv got EOF from 127.27.110.195:33901 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 07:00:59 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:59 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:59.360411 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:59.354627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:40951)
0812 07:00:59.354898 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:59.354911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:59.355048 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:59.355472 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:59.355476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:59.355753 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:59.355876 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:59.355883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.356708 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.356710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:59.357281 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:59.357289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.357751 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.357754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:59.357817 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:59.358284 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:59.358296 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:59.360237 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
W20260812 07:00:59.360469 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:59.354663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:40951 (local address 127.27.110.254:36415)
0812 07:00:59.354919 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:59.354924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:59.354989 (+ 65us) server_negotiation.cc:407] Connection header received
0812 07:00:59.355202 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:59.355205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:59.355278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:59.355386 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:59.355993 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.356599 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.357410 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.357645 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.357729 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:59.360353 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:40951: BlockingRecv error: recv got EOF from 127.27.110.193:40951 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
Aug 12 07:00:59 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:59 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:59.537880 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:59.532129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:45103)
0812 07:00:59.532557 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:59.532600 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:59.532743 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:59.532986 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:59.532988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:59.533183 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:59.533281 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:59.533286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.534088 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.534091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:59.534684 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:59.534693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.535124 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.535126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:59.535191 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:59.535692 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:59.535704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:59.537711 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":197,"threads_started":1}
W20260812 07:00:59.537922 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:59.532268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:45103 (local address 127.27.110.254:36415)
0812 07:00:59.532436 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:59.532439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:59.532695 (+ 256us) server_negotiation.cc:407] Connection header received
0812 07:00:59.532829 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:59.532831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:59.532875 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:59.532941 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:59.533384 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.533979 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.534790 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.535000 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.535124 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:59.537825 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:45103: BlockingRecv error: recv got EOF from 127.27.110.194:45103 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 07:00:59 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:00:59 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:00:59.736780 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:00:59.730655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:47203)
0812 07:00:59.731039 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 07:00:59.731059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:00:59.731198 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:00:59.731506 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:00:59.731509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:00:59.731712 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:00:59.731813 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:59.731818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.732710 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.732713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:59.733368 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:00:59.733377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.733875 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.733879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:00:59.733960 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:59.734666 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:00:59.734681 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:00:59.736663 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
W20260812 07:00:59.736958 12661 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:00:59.730747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:47203 (local address 127.27.110.254:36415)
0812 07:00:59.730911 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 07:00:59.730918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:00:59.731166 (+ 248us) server_negotiation.cc:407] Connection header received
0812 07:00:59.731288 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:00:59.731291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:00:59.731343 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:00:59.731420 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:00:59.731958 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.732566 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.733483 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:00:59.733744 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:00:59.733839 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:00:59.736796 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:47203: BlockingRecv error: recv got EOF from 127.27.110.195:47203 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 12 07:01:00 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:00 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:00.370366 12668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:00.361256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:54165)
0812 07:01:00.361574 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:00.361626 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:00.361737 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:00.361992 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:00.361994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:00.362206 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:00.362309 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:00.362315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.363182 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.363185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:00.363826 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:00.363836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.364281 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.364284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:00.364350 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:00.367917 (+ 3567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:00.367938 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:00.370237 (+ 2299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":128,"threads_started":1}
W20260812 07:01:00.370527 12669 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:00.361325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:54165 (local address 127.27.110.254:36415)
0812 07:01:00.361738 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:00.361741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:00.361757 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:01:00.361820 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:00.361822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:00.361866 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:00.361949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:00.362439 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.363013 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.363937 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.364172 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.364256 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:00.370374 (+ 6118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:54165: BlockingRecv error: recv got EOF from 127.27.110.193:54165 (error 108)
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":205,"threads_started":1}
Aug 12 07:01:00 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:00 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:00.546015 12670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:00.538863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:57593)
0812 07:01:00.539166 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:00.539187 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:00.539296 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:00.539600 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:00.539602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:00.539785 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:00.539886 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:00.539890 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.541082 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.541086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:00.542076 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:00.542097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.542843 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.542850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:00.542952 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:00.543674 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:00.543691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:00.545894 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
W20260812 07:01:00.546190 12669 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:00.538901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57593 (local address 127.27.110.254:36415)
0812 07:01:00.539092 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:00.539098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:00.539269 (+ 171us) server_negotiation.cc:407] Connection header received
0812 07:01:00.539392 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:00.539397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:00.539467 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:00.539582 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:00.540021 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.540897 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.542224 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.542637 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.542818 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:00.546035 (+ 3217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:57593: BlockingRecv error: recv got EOF from 127.27.110.194:57593 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 12 07:01:00 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:00 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:00.743957 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:00.737774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:43425)
0812 07:01:00.738100 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:00.738115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:00.738228 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:00.738537 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:00.738541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:00.738785 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:00.738933 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:00.738941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.739943 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.739948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:00.740574 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:00.740583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.741066 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.741070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:00.741161 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:00.741684 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:00.741696 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:00.743788 (+ 2092us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
W20260812 07:01:00.744055 12669 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:00.737825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:43425 (local address 127.27.110.254:36415)
0812 07:01:00.737989 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:00.737994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:00.738198 (+ 204us) server_negotiation.cc:407] Connection header received
0812 07:01:00.738316 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:00.738319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:00.738372 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:00.738451 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:00.739048 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.739826 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.740699 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:00.740941 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:00.741051 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:00.743938 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:43425: BlockingRecv error: recv got EOF from 127.27.110.195:43425 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 07:01:01 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:01 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:01.377692 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:01.371224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:50241)
0812 07:01:01.371573 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:01.371613 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:01.371733 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:01.372058 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:01.372060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:01.372243 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:01.372345 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:01.372350 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.373098 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.373101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:01.373693 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:01.373702 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.374213 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.374216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:01.374290 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:01.374951 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:01.374964 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:01.377543 (+ 2579us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":83,"threads_started":1}
W20260812 07:01:01.377854 12674 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:01.371324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:50241 (local address 127.27.110.254:36415)
0812 07:01:01.371622 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:01.371626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:01.371683 (+ 57us) server_negotiation.cc:407] Connection header received
0812 07:01:01.371873 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:01.371881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:01.371931 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:01.372017 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:01.372449 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.372986 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.373823 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.374059 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.374147 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:01.377716 (+ 3569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:50241: BlockingRecv error: recv got EOF from 127.27.110.193:50241 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
Aug 12 07:01:01 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:01 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:01.553210 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:01.547058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:54619)
0812 07:01:01.547441 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:01.547455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:01.547560 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:01.547799 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:01.547801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:01.547992 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:01.548104 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:01.548109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.548875 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.548879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:01.549700 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:01.549712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.550154 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.550157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:01.550226 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:01.550772 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:01.550785 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:01.552998 (+ 2213us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
W20260812 07:01:01.553323 12674 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:01.547189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:54619 (local address 127.27.110.254:36415)
0812 07:01:01.547352 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:01.547357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:01.547527 (+ 170us) server_negotiation.cc:407] Connection header received
0812 07:01:01.547650 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:01.547652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:01.547698 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:01.547766 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:01.548202 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.548746 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.549820 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.550036 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.550121 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:01.553188 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:54619: BlockingRecv error: recv got EOF from 127.27.110.194:54619 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 07:01:01 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:01 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:01.750715 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:01.744755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:42407)
0812 07:01:01.745161 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:01.745175 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:01.745316 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:01.745612 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:01.745615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:01.745855 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:01.745955 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:01.745960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.746762 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.746767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:01.747430 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:01.747441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.747924 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.747928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:01.748018 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:01.748541 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:01.748554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:01.750596 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":88,"threads_started":1}
W20260812 07:01:01.750937 12674 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:01.744872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:42407 (local address 127.27.110.254:36415)
0812 07:01:01.745089 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:01.745094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:01.745258 (+ 164us) server_negotiation.cc:407] Connection header received
0812 07:01:01.745429 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:01.745433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:01.745490 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:01.745582 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:01.746080 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.746646 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.747565 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:01.747801 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:01.747916 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:01.750802 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:42407: BlockingRecv error: recv got EOF from 127.27.110.195:42407 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 12 07:01:02 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:02 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:02.384356 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:02.378535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:58373)
0812 07:01:02.378830 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:02.378844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:02.378937 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:02.379207 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:02.379210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:02.379397 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:02.379501 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:02.379506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.380269 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.380272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:02.380870 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:02.380879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.381336 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.381339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:02.381405 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:02.382001 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:02.382014 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:02.384176 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
W20260812 07:01:02.384413 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:02.378720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:58373 (local address 127.27.110.254:36415)
0812 07:01:02.378953 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:02.378956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:02.378970 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:01:02.379005 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:02.379007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:02.379046 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:02.379165 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:02.379625 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.380166 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.380985 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.381223 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.381302 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:02.384294 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:58373: BlockingRecv error: recv got EOF from 127.27.110.193:58373 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":87,"threads_started":1}
Aug 12 07:01:02 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:02 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:02.560549 12679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:02.554344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:55727)
0812 07:01:02.554791 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:02.554806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:02.554950 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:02.555265 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:02.555268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:02.555520 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:02.555619 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:02.555625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.556671 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.556677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:02.557256 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:02.557264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.557752 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.557756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:02.557842 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:02.558373 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:02.558386 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:02.560415 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":108,"threads_started":1}
W20260812 07:01:02.560725 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:02.554398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:55727 (local address 127.27.110.254:36415)
0812 07:01:02.554594 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:02.554600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:02.554898 (+ 298us) server_negotiation.cc:407] Connection header received
0812 07:01:02.555062 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:02.555065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:02.555145 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:02.555240 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:02.555740 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.556555 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.557388 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.557633 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.557746 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:02.560555 (+ 2809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:55727: BlockingRecv error: recv got EOF from 127.27.110.194:55727 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 07:01:02 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:02 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:02.760443 12680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:02.751578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:43217)
0812 07:01:02.751877 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:02.751890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:02.751984 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:02.752288 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:02.752290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:02.752549 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:02.752719 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:02.752727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.753883 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.753886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:02.754446 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:02.754454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.754954 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.754960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:02.755032 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:02.755638 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:02.755650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:02.760202 (+ 4552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
W20260812 07:01:02.761034 12678 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:02.751760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:43217 (local address 127.27.110.254:36415)
0812 07:01:02.751939 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:02.751943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:02.751958 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:01:02.752078 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:02.752081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:02.752157 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:02.752264 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:02.752862 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.753776 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.754563 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:02.754798 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:02.754887 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:02.760826 (+ 5939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:43217: BlockingRecv error: recv got EOF from 127.27.110.195:43217 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 12 07:01:03 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:03 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:03.391238 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:03.385310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:47579)
0812 07:01:03.385746 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:03.385763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:03.385894 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:03.386275 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:03.386278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:03.386471 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:03.386581 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:03.386588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.387432 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.387435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:03.388023 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:03.388033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.388524 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.388527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:03.388594 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:03.389151 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:03.389163 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:03.391067 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":148,"threads_started":1}
W20260812 07:01:03.391428 12682 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:03.385414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:47579 (local address 127.27.110.254:36415)
0812 07:01:03.385694 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:03.385698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:03.385856 (+ 158us) server_negotiation.cc:407] Connection header received
0812 07:01:03.386036 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:03.386039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:03.386117 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:03.386195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:03.386723 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.387321 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.388190 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.388417 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.388503 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:03.391255 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:47579: BlockingRecv error: recv got EOF from 127.27.110.193:47579 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
Aug 12 07:01:03 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:03 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:03.567276 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:03.561361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:57529)
0812 07:01:03.561676 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:03.561694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:03.561814 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:03.562133 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:03.562136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:03.562357 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:03.562462 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:03.562468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.563312 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.563315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:03.563888 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:03.563897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.564382 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.564385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:03.564472 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:03.565051 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:03.565070 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:03.567097 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
W20260812 07:01:03.567344 12682 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:03.561428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57529 (local address 127.27.110.254:36415)
0812 07:01:03.561667 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:03.561672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:03.561753 (+ 81us) server_negotiation.cc:407] Connection header received
0812 07:01:03.561949 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:03.561952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:03.562006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:03.562093 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:03.562596 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.563200 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.564021 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.564260 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.564366 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:03.567234 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:57529: BlockingRecv error: recv got EOF from 127.27.110.194:57529 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 12 07:01:03 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:03 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:03.768901 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:03.762677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:54171)
0812 07:01:03.763031 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:03.763045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:03.763205 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:03.763479 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:03.763482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:03.763743 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:03.763839 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:03.763845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.764619 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.764625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:03.765213 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:03.765222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.765726 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.765729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:03.765816 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:03.766392 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:03.766405 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:03.768764 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
W20260812 07:01:03.769043 12682 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:03.762800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:54171 (local address 127.27.110.254:36415)
0812 07:01:03.762963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:03.762968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:03.763149 (+ 181us) server_negotiation.cc:407] Connection header received
0812 07:01:03.763315 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:03.763318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:03.763370 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:03.763457 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:03.763965 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.764494 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.765343 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:03.765606 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:03.765731 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:03.768909 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:54171: BlockingRecv error: recv got EOF from 127.27.110.195:54171 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 07:01:04 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:04 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:04.398198 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:04.392105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:41131)
0812 07:01:04.392456 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:04.392469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:04.392567 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:04.392843 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:04.392845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:04.393048 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:04.393152 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:04.393158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.393983 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.393986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:04.394612 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:04.394624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.395210 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.395214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:04.395283 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:04.395831 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:04.395843 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:04.397940 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":139,"threads_started":1}
W20260812 07:01:04.398257 12689 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:04.392202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:41131 (local address 127.27.110.254:36415)
0812 07:01:04.392463 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:04.392467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:04.392570 (+ 103us) server_negotiation.cc:407] Connection header received
0812 07:01:04.392664 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:04.392666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:04.392710 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:04.392793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:04.393271 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.393846 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.394822 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.395057 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.395173 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:04.398107 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:41131: BlockingRecv error: recv got EOF from 127.27.110.193:41131 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
Aug 12 07:01:04 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:04 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:04.574394 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:04.568183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:55037)
0812 07:01:04.568641 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:04.568659 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:04.568813 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:04.569106 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:04.569109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:04.569339 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:04.569442 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:04.569447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.570314 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.570318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:04.570975 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:04.570985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.571507 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.571511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:04.571591 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:04.572197 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:04.572209 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:04.574198 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":102,"threads_started":1}
W20260812 07:01:04.574452 12689 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:04.568315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:55037 (local address 127.27.110.254:36415)
0812 07:01:04.568630 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:04.568635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:04.568749 (+ 114us) server_negotiation.cc:407] Connection header received
0812 07:01:04.568918 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:04.568921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:04.568975 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:04.569066 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:04.569569 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.570192 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.571126 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.571384 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.571503 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:04.574334 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:55037: BlockingRecv error: recv got EOF from 127.27.110.194:55037 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
Aug 12 07:01:04 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:04 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:04.775574 12694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:04.769669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:50339)
0812 07:01:04.770015 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:04.770032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:04.770156 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:04.770466 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:04.770469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:04.770742 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:04.770842 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:04.770848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.771721 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.771726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:04.772307 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:04.772315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.772806 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.772810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:04.772900 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:04.773401 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:04.773413 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:04.775444 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":116,"threads_started":1}
W20260812 07:01:04.775781 12689 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:04.769872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:50339 (local address 127.27.110.254:36415)
0812 07:01:04.770029 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:04.770035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:04.770095 (+ 60us) server_negotiation.cc:407] Connection header received
0812 07:01:04.770270 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:04.770273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:04.770342 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:04.770428 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:04.770968 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.771608 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.772440 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:04.772683 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:04.772790 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:04.775631 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:50339: BlockingRecv error: recv got EOF from 127.27.110.195:50339 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 12 07:01:05 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:05 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:05.409915 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:05.399201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:55599)
0812 07:01:05.399580 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:05.399596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:05.399748 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:05.400058 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:05.400061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:05.400288 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:05.400392 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:05.400397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.401190 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.401196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:05.401871 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:05.401882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.402479 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.402483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:05.402572 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:05.407710 (+ 5138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:05.407725 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:05.409777 (+ 2052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
W20260812 07:01:05.410087 12697 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:05.399240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:55599 (local address 127.27.110.254:36415)
0812 07:01:05.399544 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:05.399550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:05.399692 (+ 142us) server_negotiation.cc:407] Connection header received
0812 07:01:05.399863 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:05.399867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:05.399918 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:05.400024 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:05.400520 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.401075 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.402002 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.402361 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.402458 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:05.409946 (+ 7488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:55599: BlockingRecv error: recv got EOF from 127.27.110.193:55599 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":70,"threads_started":1}
Aug 12 07:01:05 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:05 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:05.581456 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:05.575300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:55869)
0812 07:01:05.575641 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:05.575658 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:05.575808 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:05.576104 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:05.576107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:05.576332 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:05.576428 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:05.576434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.577279 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.577284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:05.577867 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:05.577876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.578345 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.578349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:05.578438 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:05.578999 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:05.579016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:05.581280 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":91,"threads_started":1}
W20260812 07:01:05.581542 12697 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:05.575437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:55869 (local address 127.27.110.254:36415)
0812 07:01:05.575638 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:05.575642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:05.575753 (+ 111us) server_negotiation.cc:407] Connection header received
0812 07:01:05.575919 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:05.575924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:05.575982 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:05.576074 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:05.576604 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.577161 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.577995 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.578224 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.578339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:05.581415 (+ 3076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:55869: BlockingRecv error: recv got EOF from 127.27.110.194:55869 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 12 07:01:05 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:05 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:05.782157 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:05.776581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:59075)
0812 07:01:05.776884 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:05.776925 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:05.777072 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:05.777300 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:05.777302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:05.777496 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:05.777595 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:05.777600 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.778330 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.778333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:05.778910 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:05.778919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.779422 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.779425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:05.779499 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:05.780127 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:05.780140 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:05.782027 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":103,"threads_started":1}
W20260812 07:01:05.782426 12697 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:05.776670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:59075 (local address 127.27.110.254:36415)
0812 07:01:05.776820 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:05.776827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:05.777049 (+ 222us) server_negotiation.cc:407] Connection header received
0812 07:01:05.777144 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:05.777147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:05.777193 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:05.777260 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:05.777697 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.778230 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.779041 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:05.779298 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:05.779389 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:05.782268 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:59075: BlockingRecv error: recv got EOF from 127.27.110.195:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 07:01:06 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:06 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:06.416735 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:06.410923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:55095)
0812 07:01:06.411228 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:06.411245 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:06.411358 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:06.411610 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:06.411612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:06.411815 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:06.411921 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:06.411927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.412659 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.412662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:06.413230 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:06.413238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.413684 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.413694 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:06.413758 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:06.414242 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:06.414254 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:06.416611 (+ 2357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
W20260812 07:01:06.416931 12708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:06.411060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:55095 (local address 127.27.110.254:36415)
0812 07:01:06.411336 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:06.411340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:06.411354 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:01:06.411444 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:06.411446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:06.411490 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:06.411580 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:06.412022 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.412551 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.413337 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.413580 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.413661 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:06.416753 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:55095: BlockingRecv error: recv got EOF from 127.27.110.193:55095 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
Aug 12 07:01:06 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:06 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:06.587791 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:06.582366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:51579)
0812 07:01:06.582679 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:06.582693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:06.582796 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:06.583035 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:06.583038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:06.583248 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:06.583349 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:06.583355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.584151 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.584153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:06.584727 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:06.584735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.585158 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.585161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:06.585221 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:06.585739 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:06.585751 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:06.587670 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
W20260812 07:01:06.587930 12708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:06.582432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:51579 (local address 127.27.110.254:36415)
0812 07:01:06.582609 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:06.582613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:06.582764 (+ 151us) server_negotiation.cc:407] Connection header received
0812 07:01:06.582875 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:06.582877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:06.582921 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:06.582996 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:06.583454 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.584038 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.584834 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.585053 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.585140 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:06.587806 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:51579: BlockingRecv error: recv got EOF from 127.27.110.194:51579 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 07:01:06 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:06 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:06.789369 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:06.783003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:40395)
0812 07:01:06.783420 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:06.783434 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:06.783609 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:06.783910 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:06.783913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:06.784141 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:06.784248 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:06.784255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.785048 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.785051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:06.785641 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:06.785649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.786123 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.786126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:06.786219 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:06.786774 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:06.786787 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:06.789181 (+ 2394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":135,"threads_started":1}
W20260812 07:01:06.789520 12708 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:06.783172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:40395 (local address 127.27.110.254:36415)
0812 07:01:06.783363 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:06.783368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:06.783513 (+ 145us) server_negotiation.cc:407] Connection header received
0812 07:01:06.783731 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:06.783735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:06.783793 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:06.783881 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:06.784370 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.784930 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.785765 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:06.785996 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:06.786117 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:06.789372 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:40395: BlockingRecv error: recv got EOF from 127.27.110.195:40395 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 12 07:01:07 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:07 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:07.423844 12712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:07.417612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:35831)
0812 07:01:07.417929 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:07.417944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:07.418048 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:07.418303 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:07.418306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:07.418532 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:07.418637 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:07.418642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.419807 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.419812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:07.420500 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:07.420511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.421009 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.421013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:07.421108 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:07.421688 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:07.421701 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:07.423723 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":110,"threads_started":1}
W20260812 07:01:07.424104 12713 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:07.417712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:35831 (local address 127.27.110.254:36415)
0812 07:01:07.418022 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:07.418032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:07.418049 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:01:07.418142 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:07.418144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:07.418187 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:07.418267 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:07.418785 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.419681 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.420626 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.420880 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.420994 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:07.423946 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:35831: BlockingRecv error: recv got EOF from 127.27.110.193:35831 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":68,"threads_started":1}
Aug 12 07:01:07 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:07 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:07.594527 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:07.588662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:54343)
0812 07:01:07.588940 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:07.588954 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:07.589060 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:07.589343 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:07.589346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:07.589549 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:07.589692 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:07.589699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.590488 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.590491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:07.591064 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:07.591103 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.591566 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.591569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:07.591629 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:07.592240 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:07.592252 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:07.594357 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
W20260812 07:01:07.594586 12713 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:07.588796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:54343 (local address 127.27.110.254:36415)
0812 07:01:07.588940 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:07.588943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:07.589027 (+ 84us) server_negotiation.cc:407] Connection header received
0812 07:01:07.589139 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:07.589141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:07.589185 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:07.589255 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:07.589825 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.590373 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.591220 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.591455 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.591551 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:07.594478 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:54343: BlockingRecv error: recv got EOF from 127.27.110.194:54343 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 12 07:01:07 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:07 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:07.798834 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:07.790338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:47661)
0812 07:01:07.790759 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:07.790776 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:07.790908 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:07.791240 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:07.791244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:07.791526 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:07.791684 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:07.791694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.792948 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.792954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:07.794155 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:07.794174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.794933 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.794939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:07.795069 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:07.795811 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:07.795832 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:07.798597 (+ 2765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":108,"threads_started":1}
W20260812 07:01:07.798970 12713 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:07.790438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:47661 (local address 127.27.110.254:36415)
0812 07:01:07.790623 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:07.790628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:07.790865 (+ 237us) server_negotiation.cc:407] Connection header received
0812 07:01:07.791009 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:07.791014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:07.791112 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:07.791209 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:07.791829 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.792801 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.794317 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:07.794788 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:07.794919 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:07.798795 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:47661: BlockingRecv error: recv got EOF from 127.27.110.195:47661 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 12 07:01:08 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:08 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:08.430615 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:08.424609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:42991)
0812 07:01:08.424931 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:08.424945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:08.425041 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:08.425291 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:08.425293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:08.425497 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:08.425641 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:08.425648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.426624 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.426628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:08.427300 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:08.427310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.427798 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.427801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:08.427873 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:08.428386 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:08.428399 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:08.430481 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":94,"threads_started":1}
W20260812 07:01:08.430801 12719 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:08.424719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:42991 (local address 127.27.110.254:36415)
0812 07:01:08.425034 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:08.425038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:08.425052 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:01:08.425129 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:08.425131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:08.425173 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:08.425248 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:08.425914 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.426475 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.427417 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.427676 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.427766 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:08.430647 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:42991: BlockingRecv error: recv got EOF from 127.27.110.193:42991 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":70,"threads_started":1}
Aug 12 07:01:08 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:08 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:08.601184 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:08.595337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:47317)
0812 07:01:08.595692 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:08.595710 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:08.595835 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:08.596138 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:08.596141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:08.596367 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:08.596466 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:08.596472 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.597253 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.597259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:08.597847 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:08.597856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.598343 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.598347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:08.598438 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:08.598967 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:08.598979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:08.601000 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":97,"threads_started":1}
W20260812 07:01:08.601243 12719 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:08.595501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:47317 (local address 127.27.110.254:36415)
0812 07:01:08.595695 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:08.595701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:08.595775 (+ 74us) server_negotiation.cc:407] Connection header received
0812 07:01:08.595948 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:08.595952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:08.596006 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:08.596101 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:08.596599 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.597134 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.597977 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.598220 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.598336 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:08.601133 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:47317: BlockingRecv error: recv got EOF from 127.27.110.194:47317 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 12 07:01:08 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:08 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.195@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:08.805542 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:08.799778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.195:36843)
0812 07:01:08.800047 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:08.800062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:08.800171 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:08.800484 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:08.800487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:08.800676 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:08.800771 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:08.800776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.801497 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.801499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:08.802066 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:08.802074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.802638 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.802641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:08.802743 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:08.803350 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:08.803371 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:08.805382 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
W20260812 07:01:08.805812 12719 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:08.799907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:36843 (local address 127.27.110.254:36415)
0812 07:01:08.800101 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:08.800105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:08.800120 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:01:08.800311 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:08.800314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:08.800364 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:08.800441 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:08.800878 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.801401 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.802172 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:08.802496 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:08.802628 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:08.805686 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.195:36843: BlockingRecv error: recv got EOF from 127.27.110.195:36843 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 07:01:09 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:09 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.193@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:09.437470 12724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:09.431636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.193:57737)
0812 07:01:09.431966 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:09.431980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:09.432082 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:09.432355 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:09.432358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:09.432593 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:09.432721 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:09.432730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.433589 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.433593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:09.434176 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:09.434184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.434653 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.434657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:09.434746 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:09.435377 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:09.435389 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:09.437296 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
W20260812 07:01:09.437547 12726 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:09.431694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:57737 (local address 127.27.110.254:36415)
0812 07:01:09.431960 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:09.431967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:09.432059 (+ 92us) server_negotiation.cc:407] Connection header received
0812 07:01:09.432163 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:09.432165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:09.432229 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:09.432332 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:09.432840 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.433472 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.434280 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.434503 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.434582 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:09.437422 (+ 2840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.193:57737: BlockingRecv error: recv got EOF from 127.27.110.193:57737 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
Aug 12 07:01:09 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 07:01:09 dist-test-slave-w4v5 krb5kdc[10939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.110.194@KRBTEST.COM for oryx/127.27.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 07:01:09.608441 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:09.602172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36415 (local address 127.27.110.194:58799)
0812 07:01:09.602475 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:09.602489 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:09.602609 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:09.602880 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:09.602883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:09.603187 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:09.603304 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:09.603311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.604141 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.604144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:09.605260 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:09.605272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.605774 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.605778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:09.605868 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:09.606389 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:09.606401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:09.608322 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.110.254:36415: Server oryx/127.27.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
W20260812 07:01:09.608626 12726 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:09.602217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:58799 (local address 127.27.110.254:36415)
0812 07:01:09.602356 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:09.602361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:09.602584 (+ 223us) server_negotiation.cc:407] Connection header received
0812 07:01:09.602696 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:09.602699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:09.602745 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:09.602846 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:09.603441 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.603993 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.605392 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.605643 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.605757 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:09.608465 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.110.194:58799: BlockingRecv error: recv got EOF from 127.27.110.194:58799 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:01:09.693118 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 11932
I20260812 07:01:09.709440 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 12198
I20260812 07:01:09.721412 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 12406
I20260812 07:01:09.734093 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 11822
2026-08-12T07:01:09Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18376 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T07:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:01:09Z Disabled control of system clock
I20260812 07:01:09.774960 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44571
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 07:01:09.888612 12732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:09.888888 12732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:09.888923 12732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:09.893105 12732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:01:09.893167 12732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:09.893181 12732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:09.893200 12732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:01:09.893215 12732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:01:09.897974 12732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:44571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:09.899097 12732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:09.900584 12732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:09.909876 12748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:09.909948 12747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:09.909941 12732 server_base.cc:1034] running on GCE node
W20260812 07:01:09.909888 12752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:09.910753 12732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:09.911954 12732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:09.913197 12732 hybrid_clock.cc:648] HybridClock initialized: now 1786518069913175 us; error 38 us; skew 500 ppm
I20260812 07:01:09.915307 12732 webserver.cc:460] Webserver started at http://127.27.110.254:40995/ using document root <none> and password file <none>
I20260812 07:01:09.915912 12732 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:09.916010 12732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:09.916255 12732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:09.918337 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "a98bc77f309f41b3a52a887f1802b082"
format_stamp: "Formatted at 2026-08-12 07:01:09 on dist-test-slave-w4v5"
server_key: "b7f50d2f13e0d6362bda3dbf58ea53fe"
server_key_iv: "090f844686b9b5968b36ad9fcb364fa0"
server_key_version: "encryptionkey@0"
I20260812 07:01:09.918907 12732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "a98bc77f309f41b3a52a887f1802b082"
format_stamp: "Formatted at 2026-08-12 07:01:09 on dist-test-slave-w4v5"
server_key: "b7f50d2f13e0d6362bda3dbf58ea53fe"
server_key_iv: "090f844686b9b5968b36ad9fcb364fa0"
server_key_version: "encryptionkey@0"
I20260812 07:01:09.922528 12732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:01:09.924965 12760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:09.926198 12732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:01:09.926348 12732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "a98bc77f309f41b3a52a887f1802b082"
format_stamp: "Formatted at 2026-08-12 07:01:09 on dist-test-slave-w4v5"
server_key: "b7f50d2f13e0d6362bda3dbf58ea53fe"
server_key_iv: "090f844686b9b5968b36ad9fcb364fa0"
server_key_version: "encryptionkey@0"
I20260812 07:01:09.926463 12732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:09.938326 12732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:09.939019 12732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:09.939284 12732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:09.946381 12732 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:44571
I20260812 07:01:09.946393 12869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:44571 every 8 connection(s)
I20260812 07:01:09.947381 12732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:01:09.950575 12873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:01:09.951416 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 12732
I20260812 07:01:09.951548 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:01:09.951901 28091 external_mini_cluster.cc:1428] Setting key 9ddf270539cafc1c01f0179572c079d4
I20260812 07:01:09.956925 12873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082: Bootstrap starting.
I20260812 07:01:09.959167 12873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082: Neither blocks nor log segments found. Creating new log.
I20260812 07:01:09.959935 12873 log.cc:824] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082: Log is configured to *not* fsync() on all Append() calls
I20260812 07:01:09.961108 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:09.953369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41792 (local address 127.27.110.254:44571)
0812 07:01:09.954031 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:09.954045 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:09.954078 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:01:09.954726 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:09.954744 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:09.955210 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:09.955535 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:09.955967 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.956889 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.957792 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:09.958256 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:09.958429 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:09.959331 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:09.959337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:09.959349 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:09.959726 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:09.959780 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:09.959800 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:09.959909 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:09.960024 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 07:01:09.960271 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":136,"threads_started":1}
I20260812 07:01:09.961979 12873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082: No bootstrap required, opened a new log
I20260812 07:01:09.965680 12873 raft_consensus.cc:348] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98bc77f309f41b3a52a887f1802b082" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44571 } }
I20260812 07:01:09.965893 12873 raft_consensus.cc:374] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:01:09.965967 12873 raft_consensus.cc:729] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98bc77f309f41b3a52a887f1802b082, State: Initialized, Role: FOLLOWER
I20260812 07:01:09.966459 12873 consensus_queue.cc:260] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98bc77f309f41b3a52a887f1802b082" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44571 } }
I20260812 07:01:09.966590 12873 raft_consensus.cc:388] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:01:09.966673 12873 raft_consensus.cc:482] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:01:09.966778 12873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:01:09.967702 12873 raft_consensus.cc:504] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98bc77f309f41b3a52a887f1802b082" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44571 } }
I20260812 07:01:09.968034 12873 leader_election.cc:302] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a98bc77f309f41b3a52a887f1802b082; no voters:
I20260812 07:01:09.968343 12873 leader_election.cc:288] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:01:09.968470 12879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:01:09.968737 12879 raft_consensus.cc:686] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [term 1 LEADER]: Becoming Leader. State: Replica: a98bc77f309f41b3a52a887f1802b082, State: Running, Role: LEADER
I20260812 07:01:09.969096 12879 consensus_queue.cc:237] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98bc77f309f41b3a52a887f1802b082" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44571 } }
I20260812 07:01:09.969489 12873 sys_catalog.cc:564] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:01:09.970745 12881 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a98bc77f309f41b3a52a887f1802b082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98bc77f309f41b3a52a887f1802b082" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44571 } } }
I20260812 07:01:09.970810 12882 sys_catalog.cc:455] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a98bc77f309f41b3a52a887f1802b082. Latest consensus state: current_term: 1 leader_uuid: "a98bc77f309f41b3a52a887f1802b082" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a98bc77f309f41b3a52a887f1802b082" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44571 } } }
I20260812 07:01:09.970878 12881 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [sys.catalog]: This master's current role is: LEADER
I20260812 07:01:09.970916 12882 sys_catalog.cc:458] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082 [sys.catalog]: This master's current role is: LEADER
I20260812 07:01:09.971431 12908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:01:09.974082 12908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:01:09.979538 12908 catalog_manager.cc:1347] Generated new cluster ID: 9c5bc74265d44f2f90736ce3af6f605e
I20260812 07:01:09.979600 12908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:01:10.004917 12908 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:01:10.005957 12908 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:01:10.014560 12908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a98bc77f309f41b3a52a887f1802b082: Generated new TSK 0
I20260812 07:01:10.015345 12908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:01:10.021857 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44571
--builtin_ntp_servers=127.27.110.212:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 07:01:10.142585 12926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:10.142869 12926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:10.142935 12926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:10.146981 12926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:10.147086 12926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:10.147176 12926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:01:10.151918 12926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:10.153002 12926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:10.154421 12926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:10.164179 12936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:10.164196 12934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:10.164218 12926 server_base.cc:1034] running on GCE node
W20260812 07:01:10.164171 12938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:10.164908 12926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:10.165480 12926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:10.166663 12926 hybrid_clock.cc:648] HybridClock initialized: now 1786518070166648 us; error 28 us; skew 500 ppm
I20260812 07:01:10.168640 12926 webserver.cc:460] Webserver started at http://127.27.110.193:38267/ using document root <none> and password file <none>
I20260812 07:01:10.169258 12926 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:10.169322 12926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:10.169627 12926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:10.171890 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "c67d29fb4ac044fdb00da2ae5d402683"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "f718c61e7c091e22953e3ca91395e952"
server_key_iv: "9606e5fea443b7c4b2d89269033adbee"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.172461 12926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "c67d29fb4ac044fdb00da2ae5d402683"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "f718c61e7c091e22953e3ca91395e952"
server_key_iv: "9606e5fea443b7c4b2d89269033adbee"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.176151 12926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260812 07:01:10.178858 12948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.180119 12926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:01:10.180264 12926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "c67d29fb4ac044fdb00da2ae5d402683"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "f718c61e7c091e22953e3ca91395e952"
server_key_iv: "9606e5fea443b7c4b2d89269033adbee"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.180382 12926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:10.194969 12926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:10.195657 12926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:10.195850 12926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:10.196429 12926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:10.197392 12926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:10.197466 12926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.197532 12926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:10.197584 12926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.206344 12926 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:43981
I20260812 07:01:10.206387 13122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:43981 every 8 connection(s)
I20260812 07:01:10.207301 12926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:01:10.209478 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 12926
I20260812 07:01:10.209615 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:01:10.209993 28091 external_mini_cluster.cc:1428] Setting key dd32ec3456233408bf14168339bfc378
I20260812 07:01:10.212306 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44571
--builtin_ntp_servers=127.27.110.212:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 07:01:10.217662 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.209155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:57753 (local address 127.27.110.254:44571)
0812 07:01:10.209328 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:10.209336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:10.210559 (+ 1223us) server_negotiation.cc:407] Connection header received
0812 07:01:10.212313 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:10.212317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:10.212372 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:10.212473 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:10.213706 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.214236 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.215587 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.215804 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.215922 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:10.217141 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:10.217145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:10.217148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:10.217168 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:10.217188 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:10.217196 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:10.217277 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:10.217440 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 07:01:10.217517 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:01:10.218547 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.209547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:44571 (local address 127.27.110.193:57753)
0812 07:01:10.210364 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:10.210423 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:10.212069 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:10.212652 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:10.212663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:10.212779 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:10.213525 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:10.213546 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.214394 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.214398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:10.215401 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:10.215412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.215965 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.215969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:10.216154 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:10.216806 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:01:10.216807 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:10.216882 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:01:10.216978 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:10.217279 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:10.217293 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:10.217300 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:01:10.217439 (+ 139us) client_negotiation.cc:240] Negotiation successful
0812 07:01:10.217730 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":117,"threads_started":1}
I20260812 07:01:10.220141 13124 heartbeater.cc:347] Connected to a master server at 127.27.110.254:44571
I20260812 07:01:10.220399 13124 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:10.220942 13124 heartbeater.cc:511] Master 127.27.110.254:44571 requested a full tablet report, sending...
I20260812 07:01:10.222762 12806 ts_manager.cc:194] Registered new tserver with Master: c67d29fb4ac044fdb00da2ae5d402683 (127.27.110.193:43981)
I20260812 07:01:10.224548 12806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.193:57753
W20260812 07:01:10.332024 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:10.332270 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:10.332305 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:10.336221 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:10.336272 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:10.336323 13130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:01:10.340854 13130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:10.341857 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:10.343222 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:10.352806 13140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:10.352815 13144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:10.352819 13141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:10.353053 13130 server_base.cc:1034] running on GCE node
I20260812 07:01:10.353498 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:10.354115 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:10.355297 13130 hybrid_clock.cc:648] HybridClock initialized: now 1786518070355284 us; error 25 us; skew 500 ppm
I20260812 07:01:10.357362 13130 webserver.cc:460] Webserver started at http://127.27.110.194:44699/ using document root <none> and password file <none>
I20260812 07:01:10.357999 13130 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:10.358106 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:10.358361 13130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:10.360734 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "4a7c82cbb7d74359a2e2e0d26751d0c1"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "f46802b3ef019ba6f13c626b5dd4fe01"
server_key_iv: "f55e2fcbb236d90927b752a2149294b2"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.361362 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "4a7c82cbb7d74359a2e2e0d26751d0c1"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "f46802b3ef019ba6f13c626b5dd4fe01"
server_key_iv: "f55e2fcbb236d90927b752a2149294b2"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.365361 13130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:01:10.368239 13154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.369737 13130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:01:10.369844 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "4a7c82cbb7d74359a2e2e0d26751d0c1"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "f46802b3ef019ba6f13c626b5dd4fe01"
server_key_iv: "f55e2fcbb236d90927b752a2149294b2"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.369933 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:10.403471 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:10.404165 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:10.404318 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:10.404872 13130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:10.405799 13130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:10.405848 13130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.405889 13130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:10.405903 13130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.414780 13130 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:42253
I20260812 07:01:10.414808 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:42253 every 8 connection(s)
I20260812 07:01:10.415722 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:01:10.420574 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 13130
I20260812 07:01:10.420724 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:01:10.421118 28091 external_mini_cluster.cc:1428] Setting key de422899c52bb18cdb16484177fed42b
I20260812 07:01:10.423530 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44571
--builtin_ntp_servers=127.27.110.212:39321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 07:01:10.430297 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.417886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:51763 (local address 127.27.110.254:44571)
0812 07:01:10.418055 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:10.418059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:10.419236 (+ 1177us) server_negotiation.cc:407] Connection header received
0812 07:01:10.421065 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:10.421068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:10.421117 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:10.421226 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:10.422446 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.422966 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.424940 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.425163 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.425255 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:10.429632 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:10.429639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:10.429645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:10.429664 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:10.429692 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:10.429707 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:10.429822 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:10.430003 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 07:01:10.430091 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:01:10.431245 13358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.418296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:44571 (local address 127.27.110.194:51763)
0812 07:01:10.419004 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:10.419052 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:10.420789 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:10.421490 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:10.421501 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:10.421618 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:10.422284 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:10.422299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.423148 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.423154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:10.424803 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:10.424820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.425337 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.425344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:10.425621 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:10.429358 (+ 3737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:01:10.429360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:10.429399 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:01:10.429463 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:10.429825 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:10.429833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:10.429846 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 07:01:10.429944 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 07:01:10.430219 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260812 07:01:10.432456 13356 heartbeater.cc:347] Connected to a master server at 127.27.110.254:44571
I20260812 07:01:10.432720 13356 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:10.433411 13356 heartbeater.cc:511] Master 127.27.110.254:44571 requested a full tablet report, sending...
I20260812 07:01:10.434556 12806 ts_manager.cc:194] Registered new tserver with Master: 4a7c82cbb7d74359a2e2e0d26751d0c1 (127.27.110.194:42253)
I20260812 07:01:10.435199 12806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.194:51763
W20260812 07:01:10.539007 13361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:10.539335 13361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:10.539402 13361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:10.543376 13361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:10.543426 13361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:10.543505 13361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:01:10.548153 13361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:10.549381 13361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:10.550781 13361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:10.560364 13369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:10.560433 13373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:10.560424 13371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:10.560701 13361 server_base.cc:1034] running on GCE node
I20260812 07:01:10.561359 13361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:10.562063 13361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:10.563190 13361 hybrid_clock.cc:648] HybridClock initialized: now 1786518070563157 us; error 43 us; skew 500 ppm
I20260812 07:01:10.565143 13361 webserver.cc:460] Webserver started at http://127.27.110.195:41899/ using document root <none> and password file <none>
I20260812 07:01:10.565721 13361 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:10.565785 13361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:10.565999 13361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:10.568118 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "1a752846a0e149dfa4cf2afa5315a5c3"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "5d2b10926605b539516c136f553c36eb"
server_key_iv: "4baac95bea2f10b4bf99323d0a50132c"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.568827 13361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "1a752846a0e149dfa4cf2afa5315a5c3"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "5d2b10926605b539516c136f553c36eb"
server_key_iv: "4baac95bea2f10b4bf99323d0a50132c"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.572989 13361 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:01:10.575630 13386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.576836 13361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:01:10.576970 13361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "1a752846a0e149dfa4cf2afa5315a5c3"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "5d2b10926605b539516c136f553c36eb"
server_key_iv: "4baac95bea2f10b4bf99323d0a50132c"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.577085 13361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:10.591043 13361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:10.591909 13361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:10.592123 13361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:10.592715 13361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:10.593639 13361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:10.593698 13361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.593767 13361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:10.593799 13361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.602844 13361 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:34035
I20260812 07:01:10.602866 13587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:34035 every 8 connection(s)
I20260812 07:01:10.603808 13361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:01:10.611220 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 13361
I20260812 07:01:10.611352 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:01:10.611634 28091 external_mini_cluster.cc:1428] Setting key 77013ab84c2f9f137b4639457f161cc1
I20260812 07:01:10.613063 12877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.605597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:56095 (local address 127.27.110.254:44571)
0812 07:01:10.605734 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:10.605738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:10.606755 (+ 1017us) server_negotiation.cc:407] Connection header received
0812 07:01:10.608238 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:10.608242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:10.608303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:10.608383 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:10.609282 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.609824 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.611005 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.611254 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.611373 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:10.612566 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:10.612570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:10.612572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:10.612585 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:10.612611 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:10.612619 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:10.612700 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:10.612861 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 07:01:10.612923 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:01:10.613965 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.605960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:44571 (local address 127.27.110.195:56095)
0812 07:01:10.606613 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:10.606652 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:10.608007 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:10.608549 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:10.608558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:10.608648 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:10.609126 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:10.609139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.609954 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.609958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:10.610844 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:10.610854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:10.611414 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:10.611418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:10.611584 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:10.612239 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:01:10.612241 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:10.612312 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:01:10.612390 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:10.612702 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:10.612714 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:10.612722 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 07:01:10.612860 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 07:01:10.613141 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
I20260812 07:01:10.615171 13588 heartbeater.cc:347] Connected to a master server at 127.27.110.254:44571
I20260812 07:01:10.615406 13588 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:10.615896 13588 heartbeater.cc:511] Master 127.27.110.254:44571 requested a full tablet report, sending...
I20260812 07:01:10.616920 12806 ts_manager.cc:194] Registered new tserver with Master: 1a752846a0e149dfa4cf2afa5315a5c3 (127.27.110.195:34035)
I20260812 07:01:10.617507 12806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.195:56095
I20260812 07:01:10.625536 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 07:01:10.628443 12877 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:10.627397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.27.110.254:44571)
0812 07:01:10.627563 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:10.627569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:10.627740 (+ 171us) server_negotiation.cc:407] Connection header received
0812 07:01:10.627789 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:10.627792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:10.627831 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:10.627902 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:10.628353 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41806: BlockingRecv error: recv got EOF from 127.0.0.1:41806 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:01:10.628700 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 12926
I20260812 07:01:10.640134 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 13130
I20260812 07:01:10.648072 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 13361
I20260812 07:01:10.666965 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 12732
2026-08-12T07:01:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (927 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T07:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:01:10Z Disabled control of system clock
I20260812 07:01:10.702255 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42989
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 07:01:10.820134 13614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:10.820370 13614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:10.820406 13614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:10.824360 13614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:01:10.824414 13614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:10.824429 13614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:10.824445 13614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:01:10.824460 13614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:01:10.829420 13614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:10.830391 13614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:10.831763 13614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:10.840345 13629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:10.840492 13614 server_base.cc:1034] running on GCE node
W20260812 07:01:10.840435 13626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:10.840400 13627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:10.841248 13614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:10.842334 13614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:10.843523 13614 hybrid_clock.cc:648] HybridClock initialized: now 1786518070843490 us; error 49 us; skew 500 ppm
I20260812 07:01:10.845372 13614 webserver.cc:460] Webserver started at http://127.27.110.254:44801/ using document root <none> and password file <none>
I20260812 07:01:10.845942 13614 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:10.846036 13614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:10.846268 13614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:10.848351 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "bd00677a012148f5aa39336c4481d165"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "ac0cd0254b3decd2e73c39f5f00490ac"
server_key_iv: "bfe3cdfa603d37fb92bebd6eb9537eb2"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.848899 13614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "bd00677a012148f5aa39336c4481d165"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "ac0cd0254b3decd2e73c39f5f00490ac"
server_key_iv: "bfe3cdfa603d37fb92bebd6eb9537eb2"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.852595 13614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:01:10.854977 13640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:10.856240 13614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:01:10.856381 13614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "bd00677a012148f5aa39336c4481d165"
format_stamp: "Formatted at 2026-08-12 07:01:10 on dist-test-slave-w4v5"
server_key: "ac0cd0254b3decd2e73c39f5f00490ac"
server_key_iv: "bfe3cdfa603d37fb92bebd6eb9537eb2"
server_key_version: "encryptionkey@0"
I20260812 07:01:10.856493 13614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:10.860980 13614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:10.861603 13614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:10.861790 13614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:10.869138 13614 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42989
I20260812 07:01:10.869153 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42989 every 8 connection(s)
I20260812 07:01:10.870049 13614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:01:10.873431 13753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:01:10.878432 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 13614
I20260812 07:01:10.878530 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:01:10.878870 28091 external_mini_cluster.cc:1428] Setting key 8626fa0f6117c6f8cd1613dfda2eba86
I20260812 07:01:10.879268 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165: Bootstrap starting.
I20260812 07:01:10.881793 13753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165: Neither blocks nor log segments found. Creating new log.
I20260812 07:01:10.882537 13753 log.cc:824] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165: Log is configured to *not* fsync() on all Append() calls
I20260812 07:01:10.884431 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:10.880327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.27.110.254:42989)
0812 07:01:10.881001 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:10.881016 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:10.881051 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:01:10.881689 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:10.881704 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:10.882102 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:10.882450 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:10.883190 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:10.883194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:10.883206 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:10.883216 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:10.883275 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:10.883291 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:10.883412 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:10.883512 (+ 100us) server_negotiation.cc:299] Negotiation successful
0812 07:01:10.883766 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":154,"threads_started":1}
I20260812 07:01:10.884824 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165: No bootstrap required, opened a new log
I20260812 07:01:10.888612 13753 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00677a012148f5aa39336c4481d165" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42989 } }
I20260812 07:01:10.888851 13753 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:01:10.888931 13753 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd00677a012148f5aa39336c4481d165, State: Initialized, Role: FOLLOWER
I20260812 07:01:10.889346 13753 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00677a012148f5aa39336c4481d165" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42989 } }
I20260812 07:01:10.889497 13753 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:01:10.889580 13753 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:01:10.889701 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:01:10.890614 13753 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00677a012148f5aa39336c4481d165" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42989 } }
I20260812 07:01:10.890949 13753 leader_election.cc:302] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd00677a012148f5aa39336c4481d165; no voters:
I20260812 07:01:10.891317 13753 leader_election.cc:288] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:01:10.891417 13763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:01:10.891640 13763 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [term 1 LEADER]: Becoming Leader. State: Replica: bd00677a012148f5aa39336c4481d165, State: Running, Role: LEADER
I20260812 07:01:10.891983 13763 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00677a012148f5aa39336c4481d165" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42989 } }
I20260812 07:01:10.892597 13753 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:01:10.893684 13772 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd00677a012148f5aa39336c4481d165. Latest consensus state: current_term: 1 leader_uuid: "bd00677a012148f5aa39336c4481d165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00677a012148f5aa39336c4481d165" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42989 } } }
I20260812 07:01:10.893826 13772 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [sys.catalog]: This master's current role is: LEADER
I20260812 07:01:10.894274 13782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:01:10.894536 13768 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd00677a012148f5aa39336c4481d165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00677a012148f5aa39336c4481d165" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42989 } } }
I20260812 07:01:10.894667 13768 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165 [sys.catalog]: This master's current role is: LEADER
I20260812 07:01:10.897560 13782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:01:10.903649 13782 catalog_manager.cc:1347] Generated new cluster ID: 0bd0ea0bf69747feb87ec3fce5f0fe3c
I20260812 07:01:10.903735 13782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:01:10.918529 13782 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:01:10.918838 13782 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:01:10.923857 13782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd00677a012148f5aa39336c4481d165: Generated new TSK 0
I20260812 07:01:10.924511 13782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:01:10.928900 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42989
--builtin_ntp_servers=127.27.110.212:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 07:01:11.052199 13795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:11.052452 13795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:11.052493 13795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:11.056488 13795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:11.056540 13795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:11.056599 13795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:01:11.061441 13795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:11.062474 13795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:11.063910 13795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:11.073707 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:11.073827 13795 server_base.cc:1034] running on GCE node
W20260812 07:01:11.073742 13808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:11.073685 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:11.074379 13795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:11.075029 13795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:11.076236 13795 hybrid_clock.cc:648] HybridClock initialized: now 1786518071076216 us; error 32 us; skew 500 ppm
I20260812 07:01:11.078133 13795 webserver.cc:460] Webserver started at http://127.27.110.193:32777/ using document root <none> and password file <none>
I20260812 07:01:11.078774 13795 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:11.078874 13795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:11.079159 13795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:11.081197 13795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "eb8a213e62d94b6d889cd2459c286598"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "94afdfbdf48b91a87fb9a4ad55fecadb"
server_key_iv: "552f60a379dee15844db0eb0cba6947a"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.081761 13795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "eb8a213e62d94b6d889cd2459c286598"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "94afdfbdf48b91a87fb9a4ad55fecadb"
server_key_iv: "552f60a379dee15844db0eb0cba6947a"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.085415 13795 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260812 07:01:11.088027 13815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.089253 13795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:01:11.089402 13795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "eb8a213e62d94b6d889cd2459c286598"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "94afdfbdf48b91a87fb9a4ad55fecadb"
server_key_iv: "552f60a379dee15844db0eb0cba6947a"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.089515 13795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:11.094333 13795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:11.094918 13795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:11.095180 13795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:11.095717 13795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:11.096616 13795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:11.096684 13795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.096747 13795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:11.096777 13795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.105793 13795 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:42453
I20260812 07:01:11.105918 13988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:42453 every 8 connection(s)
I20260812 07:01:11.106746 13795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:01:11.107568 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 13795
I20260812 07:01:11.107693 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:01:11.107991 28091 external_mini_cluster.cc:1428] Setting key be85f597dea1bb8255938e877fd4e0f1
I20260812 07:01:11.110255 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42989
--builtin_ntp_servers=127.27.110.212:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 07:01:11.113520 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:11.108650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:52421 (local address 127.27.110.254:42989)
0812 07:01:11.108782 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:11.108787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:11.110117 (+ 1330us) server_negotiation.cc:407] Connection header received
0812 07:01:11.111584 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:11.111591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:11.111639 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:11.111737 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:11.112962 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:11.112968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:11.112974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:11.112990 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:11.113018 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:11.113028 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:11.113135 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:11.113309 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 07:01:11.113359 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:01:11.114298 13991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:11.109167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42989 (local address 127.27.110.193:52421)
0812 07:01:11.109930 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:11.109996 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:11.111355 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:11.111923 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:11.111930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:11.112016 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:11.112669 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:01:11.112671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:11.112710 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:01:11.112779 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:11.113139 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:11.113148 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:11.113154 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 07:01:11.113332 (+ 178us) client_negotiation.cc:240] Negotiation successful
0812 07:01:11.113567 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":162,"threads_started":1}
I20260812 07:01:11.115661 13989 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42989
I20260812 07:01:11.115903 13989 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:11.116312 13989 heartbeater.cc:511] Master 127.27.110.254:42989 requested a full tablet report, sending...
I20260812 07:01:11.117786 13671 ts_manager.cc:194] Registered new tserver with Master: eb8a213e62d94b6d889cd2459c286598 (127.27.110.193:42453)
W20260812 07:01:11.223892 13992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:11.224188 13992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:11.224261 13992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:11.228392 13992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:11.228466 13992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:11.228559 13992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:01:11.233349 13992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:11.234431 13992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:11.235905 13992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:11.245779 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:11.245878 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:11.245850 13992 server_base.cc:1034] running on GCE node
W20260812 07:01:11.245766 14008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:11.246444 13992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:11.247128 13992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:11.248322 13992 hybrid_clock.cc:648] HybridClock initialized: now 1786518071248306 us; error 28 us; skew 500 ppm
I20260812 07:01:11.250126 13992 webserver.cc:460] Webserver started at http://127.27.110.194:45425/ using document root <none> and password file <none>
I20260812 07:01:11.250716 13992 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:11.250780 13992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:11.250995 13992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:11.253072 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "a11136b874354bdcae2cedf9ff8ed872"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "daccf6ffcc1a7835f5159cc9a9d1d5c6"
server_key_iv: "4ad701fe1c4ba4d3587a7dba38d209ed"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.253647 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "a11136b874354bdcae2cedf9ff8ed872"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "daccf6ffcc1a7835f5159cc9a9d1d5c6"
server_key_iv: "4ad701fe1c4ba4d3587a7dba38d209ed"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.257452 13992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:01:11.260335 14024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.262075 13992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:01:11.262214 13992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "a11136b874354bdcae2cedf9ff8ed872"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "daccf6ffcc1a7835f5159cc9a9d1d5c6"
server_key_iv: "4ad701fe1c4ba4d3587a7dba38d209ed"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.262341 13992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:11.267256 13992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:11.267899 13992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:11.268045 13992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:11.268572 13992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:11.269523 13992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:11.269593 13992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.269642 13992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:11.269672 13992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.279556 13992 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:35401
I20260812 07:01:11.279577 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:35401 every 8 connection(s)
I20260812 07:01:11.280510 13992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:01:11.286903 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:11.282496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:55301 (local address 127.27.110.254:42989)
0812 07:01:11.282661 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:11.282664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:11.283744 (+ 1080us) server_negotiation.cc:407] Connection header received
0812 07:01:11.285100 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:11.285103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:11.285150 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:11.285253 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:11.286393 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:11.286396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:11.286399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:11.286412 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:11.286443 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:11.286452 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:11.286550 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:11.286714 (+ 164us) server_negotiation.cc:299] Negotiation successful
0812 07:01:11.286765 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:01:11.287859 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:11.282853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42989 (local address 127.27.110.194:55301)
0812 07:01:11.283581 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:11.283625 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:11.284882 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:11.285425 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:11.285432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:11.285523 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:11.286126 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:01:11.286128 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:11.286168 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:01:11.286236 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:11.286544 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:11.286553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:11.286562 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 07:01:11.286747 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 07:01:11.286989 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":177,"threads_started":1}
I20260812 07:01:11.288048 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 13992
I20260812 07:01:11.288169 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:01:11.288511 28091 external_mini_cluster.cc:1428] Setting key f0e6dcd5e630521fdf3fb6e383fbffec
I20260812 07:01:11.289152 14209 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42989
I20260812 07:01:11.289388 14209 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:11.289848 14209 heartbeater.cc:511] Master 127.27.110.254:42989 requested a full tablet report, sending...
I20260812 07:01:11.290729 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42989
--builtin_ntp_servers=127.27.110.212:40411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 07:01:11.290969 13671 ts_manager.cc:194] Registered new tserver with Master: a11136b874354bdcae2cedf9ff8ed872 (127.27.110.194:35401)
W20260812 07:01:11.406124 14212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:11.406376 14212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:11.406415 14212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:11.410584 14212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:11.410650 14212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:11.410706 14212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:01:11.415534 14212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:11.416700 14212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:11.418165 14212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:11.427927 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:11.427990 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:11.427922 14222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:11.428164 14212 server_base.cc:1034] running on GCE node
I20260812 07:01:11.428678 14212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:11.429329 14212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:11.430508 14212 hybrid_clock.cc:648] HybridClock initialized: now 1786518071430494 us; error 24 us; skew 500 ppm
I20260812 07:01:11.432512 14212 webserver.cc:460] Webserver started at http://127.27.110.195:35015/ using document root <none> and password file <none>
I20260812 07:01:11.433091 14212 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:11.433156 14212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:11.433393 14212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:11.435531 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "1a3b19a44f974efe803ba4a1ea36f6fa"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "9b2961c90dcfa0f674318bf69b0e3fe5"
server_key_iv: "4dcfd5d411cc1e5a419d808e8ca31df1"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.436115 14212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "1a3b19a44f974efe803ba4a1ea36f6fa"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "9b2961c90dcfa0f674318bf69b0e3fe5"
server_key_iv: "4dcfd5d411cc1e5a419d808e8ca31df1"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.439699 14212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:01:11.442186 14229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.443459 14212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:01:11.443557 14212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "1a3b19a44f974efe803ba4a1ea36f6fa"
format_stamp: "Formatted at 2026-08-12 07:01:11 on dist-test-slave-w4v5"
server_key: "9b2961c90dcfa0f674318bf69b0e3fe5"
server_key_iv: "4dcfd5d411cc1e5a419d808e8ca31df1"
server_key_version: "encryptionkey@0"
I20260812 07:01:11.443641 14212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:11.448086 14212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:11.448637 14212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:11.448771 14212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:11.449239 14212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:11.450073 14212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:11.450120 14212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.450157 14212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:11.450172 14212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:11.459295 14212 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:39601
I20260812 07:01:11.459324 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:39601 every 8 connection(s)
I20260812 07:01:11.460253 14212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:01:11.466488 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:11.462083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:42923 (local address 127.27.110.254:42989)
0812 07:01:11.462233 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:11.462237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:11.463374 (+ 1137us) server_negotiation.cc:407] Connection header received
0812 07:01:11.464731 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:11.464735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:11.464778 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:11.464864 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:11.465961 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:11.465965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:11.465967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:01:11.465986 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:11.466006 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:01:11.466014 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:11.466136 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:11.466284 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 07:01:11.466331 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:01:11.467355 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:11.462507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42989 (local address 127.27.110.195:42923)
0812 07:01:11.463231 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:11.463272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:11.464516 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:11.465036 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:11.465042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:11.465129 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:11.465710 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:01:11.465711 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:11.465750 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:01:11.465815 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:11.466121 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:11.466131 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:11.466136 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 07:01:11.466316 (+ 180us) client_negotiation.cc:240] Negotiation successful
0812 07:01:11.466545 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260812 07:01:11.468461 14421 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42989
I20260812 07:01:11.468564 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 14212
I20260812 07:01:11.468698 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:01:11.468748 14421 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:11.469041 28091 external_mini_cluster.cc:1428] Setting key b1034be327e58adc5e1ba1dcb12415cf
I20260812 07:01:11.469208 14421 heartbeater.cc:511] Master 127.27.110.254:42989 requested a full tablet report, sending...
I20260812 07:01:11.470335 13671 ts_manager.cc:194] Registered new tserver with Master: 1a3b19a44f974efe803ba4a1ea36f6fa (127.27.110.195:39601)
I20260812 07:01:11.471252 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 07:01:11.473922 13757 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:01:11.473115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.27.110.254:42989)
0812 07:01:11.473206 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:11.473210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:11.473440 (+ 230us) server_negotiation.cc:407] Connection header received
0812 07:01:11.473511 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:11.473513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:11.473542 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:11.473612 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:11.473808 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39390: BlockingRecv error: recv got EOF from 127.0.0.1:39390 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 07:01:11.474061 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 13795
I20260812 07:01:11.487416 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 13992
I20260812 07:01:11.519148 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 14212
I20260812 07:01:11.539734 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 13614
2026-08-12T07:01:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (875 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:01:11 dist-test-slave-w4v5 krb5kdc[14445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:01:11 dist-test-slave-w4v5 krb5kdc[14445](info): set up 2 sockets
Aug 12 07:01:11 dist-test-slave-w4v5 krb5kdc[14445](info): commencing operation
krb5kdc: starting...
W20260812 07:01:12.846769 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:01:12 dist-test-slave-w4v5 krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:01:12.978116 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:37973
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:37973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:01:13.090020 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:13.090343 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:13.090425 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:13.095144 14463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:01:13.095232 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:13.095271 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:13.095310 14463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:01:13.095345 14463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:01:13.100664 14463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:37973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:37973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:13.101761 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:13.102813 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:13.111899 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.111958 14472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.111912 14481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:13.112268 14463 server_base.cc:1034] running on GCE node
I20260812 07:01:13.112975 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:13.114231 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:13.115442 14463 hybrid_clock.cc:648] HybridClock initialized: now 1786518073115410 us; error 47 us; skew 500 ppm
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:01:13.118427 14463 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:01:13.119415 14463 webserver.cc:460] Webserver started at http://127.27.110.254:44545/ using document root <none> and password file <none>
I20260812 07:01:13.119982 14463 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:13.120078 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:13.120318 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:13.122382 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "291ca746fd5b4a5fb9d671d8138a0240"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "18d216eff06323273bec62ee02e1cba6"
server_key_iv: "f9230503f57ca805082e322454981f28"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.122956 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "291ca746fd5b4a5fb9d671d8138a0240"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "18d216eff06323273bec62ee02e1cba6"
server_key_iv: "f9230503f57ca805082e322454981f28"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.126761 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:01:13.129236 14495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.130478 14463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:01:13.130631 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "291ca746fd5b4a5fb9d671d8138a0240"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "18d216eff06323273bec62ee02e1cba6"
server_key_iv: "f9230503f57ca805082e322454981f28"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.130743 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:13.144302 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:13.147554 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:13.147809 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:13.155514 14463 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:37973
I20260812 07:01:13.155803 14589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:37973 every 8 connection(s)
I20260812 07:01:13.156545 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:01:13.159431 14590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:01:13.164790 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 14463
I20260812 07:01:13.164942 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:01:13.165067 14590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: Bootstrap starting.
I20260812 07:01:13.165277 28091 external_mini_cluster.cc:1428] Setting key 32f83cc5da49090d11c648c428cbe18c
I20260812 07:01:13.167739 14590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: Neither blocks nor log segments found. Creating new log.
I20260812 07:01:13.168640 14590 log.cc:824] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: Log is configured to *not* fsync() on all Append() calls
I20260812 07:01:13.170683 14590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: No bootstrap required, opened a new log
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:01:13.175158 14590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291ca746fd5b4a5fb9d671d8138a0240" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37973 } }
I20260812 07:01:13.175403 14590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:01:13.175493 14590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291ca746fd5b4a5fb9d671d8138a0240, State: Initialized, Role: FOLLOWER
I20260812 07:01:13.175982 14590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291ca746fd5b4a5fb9d671d8138a0240" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37973 } }
I20260812 07:01:13.176142 14590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:01:13.176232 14590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:01:13.176335 14590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:01:13.177232 14590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291ca746fd5b4a5fb9d671d8138a0240" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37973 } }
I20260812 07:01:13.177578 14590 leader_election.cc:302] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 291ca746fd5b4a5fb9d671d8138a0240; no voters:
I20260812 07:01:13.177945 14590 leader_election.cc:288] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:01:13.177985 14603 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:01:13.178186 14603 raft_consensus.cc:686] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [term 1 LEADER]: Becoming Leader. State: Replica: 291ca746fd5b4a5fb9d671d8138a0240, State: Running, Role: LEADER
I20260812 07:01:13.178561 14603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291ca746fd5b4a5fb9d671d8138a0240" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37973 } }
I20260812 07:01:13.179287 14590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:01:13.180943 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.166755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57208 (local address 127.27.110.254:37973)
0812 07:01:13.167211 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.167236 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.167279 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:01:13.168025 (+ 746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.168045 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.168392 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.168672 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:13.169583 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.170582 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.171788 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.172269 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.172400 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.175432 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:13.175439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:13.175457 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:01:13.175491 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:13.177445 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.177977 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.177983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.177989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.178140 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.178526 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.178528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.178529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.178994 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:13.179272 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.179555 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.179905 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":138,"threads_started":1}
I20260812 07:01:13.182010 14605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "291ca746fd5b4a5fb9d671d8138a0240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291ca746fd5b4a5fb9d671d8138a0240" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37973 } } }
I20260812 07:01:13.182097 14604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291ca746fd5b4a5fb9d671d8138a0240. Latest consensus state: current_term: 1 leader_uuid: "291ca746fd5b4a5fb9d671d8138a0240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "291ca746fd5b4a5fb9d671d8138a0240" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37973 } } }
I20260812 07:01:13.182154 14605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [sys.catalog]: This master's current role is: LEADER
I20260812 07:01:13.182159 14604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240 [sys.catalog]: This master's current role is: LEADER
W20260812 07:01:13.184290 14628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:01:13.184439 14628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:01:13.184548 14631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:01:13.185472 14631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:01:13.190971 14631 catalog_manager.cc:1347] Generated new cluster ID: 5a3a57e713f042ceab898870bc986811
I20260812 07:01:13.191042 14631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:01:13.200114 14631 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:01:13.201009 14631 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:01:13.207353 14631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: Generated new TSK 0
I20260812 07:01:13.208034 14631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:01:13.242465 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37973
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:01:13.359737 14641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:13.359983 14641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:13.360033 14641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:13.363927 14641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:13.363979 14641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:13.364039 14641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:01:13.369130 14641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:13.370044 14641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:13.371001 14641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:13.380589 14649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.380698 14654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:13.380621 14641 server_base.cc:1034] running on GCE node
W20260812 07:01:13.380618 14652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:13.381273 14641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:13.381886 14641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:13.383059 14641 hybrid_clock.cc:648] HybridClock initialized: now 1786518073383026 us; error 43 us; skew 500 ppm
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:01:13.386147 14641 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:01:13.386979 14641 webserver.cc:460] Webserver started at http://127.27.110.193:43719/ using document root <none> and password file <none>
I20260812 07:01:13.387559 14641 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:13.387621 14641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:13.387777 14641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:13.389731 14641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "5e3dc8b2d114418187f7d0d759600573"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "44cb1035d7c746bafcf1a8113d6ea96a"
server_key_iv: "09dc4ee8a43960497362cda1a2983c65"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.390230 14641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "5e3dc8b2d114418187f7d0d759600573"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "44cb1035d7c746bafcf1a8113d6ea96a"
server_key_iv: "09dc4ee8a43960497362cda1a2983c65"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.393865 14641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:01:13.396333 14663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.397509 14641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:01:13.397699 14641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "5e3dc8b2d114418187f7d0d759600573"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "44cb1035d7c746bafcf1a8113d6ea96a"
server_key_iv: "09dc4ee8a43960497362cda1a2983c65"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.397789 14641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:13.416355 14641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:13.419658 14641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:13.419809 14641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:13.420322 14641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:13.421084 14641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:13.421130 14641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.421168 14641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:13.421191 14641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.430058 14641 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:40195
I20260812 07:01:13.430089 14864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:40195 every 8 connection(s)
I20260812 07:01:13.430902 14641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:01:13.439963 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 14641
I20260812 07:01:13.440169 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:01:13.440491 28091 external_mini_cluster.cc:1428] Setting key 6ee13a1ffded6c90d6db823b17448340
I20260812 07:01:13.444993 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.432710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:56891 (local address 127.27.110.254:37973)
0812 07:01:13.432853 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.432858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.433651 (+ 793us) server_negotiation.cc:407] Connection header received
0812 07:01:13.435224 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.435227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.435275 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.435350 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:13.436500 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.437035 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.438150 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.438372 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.438452 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.441065 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:13.441068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:13.441070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:01:13.441094 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:13.442698 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.443281 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.443287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.443292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.443389 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.443724 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.443730 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.443735 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.443997 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:13.444101 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.444776 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.444872 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:01:13.446134 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.433078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37973 (local address 127.27.110.193:56891)
0812 07:01:13.433500 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:13.433544 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:13.434945 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:13.435510 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:13.435516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:13.435848 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:13.436331 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.436344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.437156 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.437159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.438035 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.438046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.438542 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.438554 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.438698 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.439317 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:13.439333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:13.440888 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:13.442828 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:01:13.442834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:01:13.442846 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:01:13.443182 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:01:13.443485 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:01:13.443487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:01:13.443488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:01:13.443622 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:01:13.444142 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:13.444162 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:13.444498 (+ 336us) client_negotiation.cc:769] Sending connection context
0812 07:01:13.444712 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 07:01:13.445067 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260812 07:01:13.447500 14866 heartbeater.cc:347] Connected to a master server at 127.27.110.254:37973
I20260812 07:01:13.447757 14866 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:13.448256 14866 heartbeater.cc:511] Master 127.27.110.254:37973 requested a full tablet report, sending...
I20260812 07:01:13.449718 14524 ts_manager.cc:194] Registered new tserver with Master: 5e3dc8b2d114418187f7d0d759600573 (127.27.110.193:40195)
I20260812 07:01:13.450845 14524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:56891
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:01:13.469558 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37973
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:01:13.582069 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:13.582306 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:13.582346 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:13.586467 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:13.586521 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:13.586609 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:01:13.591853 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:13.592823 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:13.593827 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:01:13.603637 14872 server_base.cc:1034] running on GCE node
W20260812 07:01:13.603590 14884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.603592 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.603586 14891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:13.604262 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:13.604907 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:13.606079 14872 hybrid_clock.cc:648] HybridClock initialized: now 1786518073606070 us; error 28 us; skew 500 ppm
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:01:13.608927 14872 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:01:13.609823 14872 webserver.cc:460] Webserver started at http://127.27.110.194:43821/ using document root <none> and password file <none>
I20260812 07:01:13.610383 14872 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:13.610496 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:13.610739 14872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:13.612782 14872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "10495a5015614b1abbb0851ddb9090fe"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "67ce962889837cf61ae4e1bc6ec8e6ff"
server_key_iv: "c23bdf3af2998928f666f0710058d4c1"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.613359 14872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "10495a5015614b1abbb0851ddb9090fe"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "67ce962889837cf61ae4e1bc6ec8e6ff"
server_key_iv: "c23bdf3af2998928f666f0710058d4c1"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.617059 14872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:01:13.619519 14903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.620721 14872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:01:13.620858 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "10495a5015614b1abbb0851ddb9090fe"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "67ce962889837cf61ae4e1bc6ec8e6ff"
server_key_iv: "c23bdf3af2998928f666f0710058d4c1"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.620968 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:13.640532 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:13.643429 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:13.643628 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:13.644212 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:13.645057 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:13.645128 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.645197 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:13.645251 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.654068 14872 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:41815
I20260812 07:01:13.654115 15094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:41815 every 8 connection(s)
I20260812 07:01:13.654949 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:01:13.656044 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 14872
I20260812 07:01:13.656175 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:01:13.656510 28091 external_mini_cluster.cc:1428] Setting key 4de4bc02a3a956dc30cecb9644e2ccd5
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:01:13.670311 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.657020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:49793 (local address 127.27.110.254:37973)
0812 07:01:13.657201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.657205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.658202 (+ 997us) server_negotiation.cc:407] Connection header received
0812 07:01:13.660261 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.660263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.660310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.660383 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:13.661624 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.662277 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.663567 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.663800 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.663909 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.666625 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:13.666628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:13.666630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:01:13.666655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:13.668207 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.668734 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.668737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.668739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.668865 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.669199 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.669201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.669202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.669380 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:13.669457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.670069 (+ 612us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.670187 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:01:13.671319 15100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.657456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37973 (local address 127.27.110.194:49793)
0812 07:01:13.658004 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:13.658080 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:13.660021 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:13.660610 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:13.660623 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:13.660990 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:13.661471 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.661485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.662410 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.662414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.663436 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.663446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.663965 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.663969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.664139 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.664779 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:13.664796 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:13.666451 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:13.668331 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:01:13.668336 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:01:13.668347 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:01:13.668605 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:01:13.668964 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:01:13.668966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:01:13.668967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:01:13.669103 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:01:13.669490 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:13.669500 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:13.669824 (+ 324us) client_negotiation.cc:769] Sending connection context
0812 07:01:13.669988 (+ 164us) client_negotiation.cc:240] Negotiation successful
0812 07:01:13.670246 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":156,"threads_started":1}
I20260812 07:01:13.672503 15098 heartbeater.cc:347] Connected to a master server at 127.27.110.254:37973
I20260812 07:01:13.672763 15098 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:13.673398 15098 heartbeater.cc:511] Master 127.27.110.254:37973 requested a full tablet report, sending...
I20260812 07:01:13.674518 14524 ts_manager.cc:194] Registered new tserver with Master: 10495a5015614b1abbb0851ddb9090fe (127.27.110.194:41815)
I20260812 07:01:13.675158 14524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:49793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:01:13.688323 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37973
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:01:13.803725 15105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:01:13.803972 15105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:01:13.804014 15105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:01:13.808043 15105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:01:13.808099 15105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:01:13.808156 15105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:01:13.813315 15105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:37973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:01:13.814283 15105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:01:13.815363 15105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:01:13.825078 15113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.825132 15112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:01:13.825038 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:01:13.825433 15105 server_base.cc:1034] running on GCE node
I20260812 07:01:13.825855 15105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:01:13.826511 15105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:01:13.827709 15105 hybrid_clock.cc:648] HybridClock initialized: now 1786518073827693 us; error 28 us; skew 500 ppm
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:01:13.830549 15105 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:01:13.831571 15105 webserver.cc:460] Webserver started at http://127.27.110.195:36049/ using document root <none> and password file <none>
I20260812 07:01:13.832137 15105 fs_manager.cc:359] Metadata directory not provided
I20260812 07:01:13.832232 15105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:01:13.832461 15105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:01:13.834589 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "b1a03a46083e4d07a6548ae5d8106358"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "57aee2c7808cccb15f5d1b538b8f8089"
server_key_iv: "89a3a96d1952da5b1055c0d4529aaef5"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.835250 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "b1a03a46083e4d07a6548ae5d8106358"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "57aee2c7808cccb15f5d1b538b8f8089"
server_key_iv: "89a3a96d1952da5b1055c0d4529aaef5"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.838979 15105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:01:13.841480 15133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.842665 15105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:01:13.842805 15105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "b1a03a46083e4d07a6548ae5d8106358"
format_stamp: "Formatted at 2026-08-12 07:01:13 on dist-test-slave-w4v5"
server_key: "57aee2c7808cccb15f5d1b538b8f8089"
server_key_iv: "89a3a96d1952da5b1055c0d4529aaef5"
server_key_version: "encryptionkey@0"
I20260812 07:01:13.842913 15105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:01:13.859541 15105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:01:13.862473 15105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:01:13.862685 15105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:01:13.863348 15105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:01:13.864164 15105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:01:13.864238 15105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.864308 15105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:01:13.864362 15105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:01:13.873436 15105 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:37083
I20260812 07:01:13.873469 15326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:37083 every 8 connection(s)
I20260812 07:01:13.874315 15105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:01:13.875835 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 15105
I20260812 07:01:13.875980 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:01:13.876413 28091 external_mini_cluster.cc:1428] Setting key 7d84c8edaaa6e69b75773179a1a5aaa3
Aug 12 07:01:13 dist-test-slave-w4v5 krb5kdc[14445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518073, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:01:13.889150 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.876216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:48343 (local address 127.27.110.254:37973)
0812 07:01:13.876405 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.876409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.877526 (+ 1117us) server_negotiation.cc:407] Connection header received
0812 07:01:13.879232 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.879235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.879282 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.879358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:13.880765 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.881358 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.882582 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.882810 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.882894 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.885393 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:13.885397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:13.885400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:01:13.885423 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:13.886898 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.887562 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.887565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.887567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.887646 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.888013 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.888016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.888018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.888222 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:13.888336 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.888842 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.888967 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:01:13.890105 15329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.876666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37973 (local address 127.27.110.195:48343)
0812 07:01:13.877384 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:13.877440 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:13.878974 (+ 1534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:13.879598 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:13.879605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:13.880055 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:01:13.880608 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.880624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.881539 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.881544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.882464 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.882474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.882968 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.882972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.883206 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.883750 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:01:13.883766 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:01:13.885215 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:01:13.887025 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:01:13.887031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:01:13.887045 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:01:13.887448 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:01:13.887754 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:01:13.887757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:01:13.887759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:01:13.887903 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:01:13.888334 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:01:13.888341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:01:13.888590 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 07:01:13.888781 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 07:01:13.889094 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":152,"threads_started":1}
I20260812 07:01:13.891294 15327 heartbeater.cc:347] Connected to a master server at 127.27.110.254:37973
I20260812 07:01:13.891507 15327 heartbeater.cc:464] Registering TS with master...
I20260812 07:01:13.891943 15327 heartbeater.cc:511] Master 127.27.110.254:37973 requested a full tablet report, sending...
I20260812 07:01:13.892925 14524 ts_manager.cc:194] Registered new tserver with Master: b1a03a46083e4d07a6548ae5d8106358 (127.27.110.195:37083)
I20260812 07:01:13.893600 14524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:48343
I20260812 07:01:13.902511 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:01:13.911996 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.904458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.27.110.254:37973)
0812 07:01:13.904620 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.904625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.904795 (+ 170us) server_negotiation.cc:407] Connection header received
0812 07:01:13.904946 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.904949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.905008 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.905092 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:01:13.905832 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.906346 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.907261 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.907493 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.907606 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.908947 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:01:13.908951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:01:13.908953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:01:13.908982 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:01:13.910226 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.910710 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.910714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.910716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.910801 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:01:13.911091 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:01:13.911097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:01:13.911102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:01:13.911328 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:01:13.911479 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.911715 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.911830 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:01:13.914606 14524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:01:13.917589 14524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:01:13.931845 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.926599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.27.110.193:40195)
0812 07:01:13.927192 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.927208 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.927234 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:01:13.927315 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.927321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.927517 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.927799 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:01:13.928016 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.928812 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.930304 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.931135 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.931267 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.931406 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.931586 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.931689 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":110,"threads_started":1}
I20260812 07:01:13.931893 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.926387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40195 (local address 127.0.0.1:39956)
0812 07:01:13.926840 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:13.926853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:13.926924 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:13.927687 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:13.927690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:13.927722 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:01:13.927907 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.927914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.928955 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.928960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.930173 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.930183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.931247 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.931251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.931361 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.931404 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 07:01:13.931523 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 07:01:13.931657 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":45,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260812 07:01:13.932142 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.925790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:37083 (local address 127.0.0.1:58100)
0812 07:01:13.926386 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:13.926401 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:13.926523 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:13.927687 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:13.927690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:13.927725 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:01:13.928040 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.928049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.929556 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.929560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.930686 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.930693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.931722 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.931726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.931822 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.931878 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 07:01:13.931934 (+ 56us) client_negotiation.cc:240] Negotiation successful
0812 07:01:13.932009 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":184,"spinlock_wait_cycles":15616,"thread_start_us":92,"threads_started":1}
I20260812 07:01:13.932343 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.925623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:41815 (local address 127.0.0.1:43454)
0812 07:01:13.926208 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:13.926262 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:13.926918 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:13.927534 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:13.927541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:13.927563 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:01:13.927733 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.927741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.929250 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.929253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.930497 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:13.930508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.931835 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.931840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:13.932004 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.932041 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 07:01:13.932130 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 07:01:13.932207 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":25,"thread_start_us":93,"threads_started":1}
I20260812 07:01:13.932773 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.926870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.27.110.195:37083)
0812 07:01:13.927291 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.927307 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.927325 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:01:13.927386 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.927390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.927531 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.927839 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:01:13.928185 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.929418 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.930796 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.931592 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.932395 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.932543 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.932591 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.932658 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":207,"threads_started":1}
I20260812 07:01:13.933396 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:13.925772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.27.110.194:41815)
0812 07:01:13.926245 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:13.926267 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:13.926699 (+ 432us) server_negotiation.cc:407] Connection header received
0812 07:01:13.927169 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:13.927175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:13.927374 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:13.927546 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:01:13.927904 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.929101 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.930639 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:13.931520 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:13.931644 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:13.931765 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:13.933138 (+ 1373us) server_negotiation.cc:299] Negotiation successful
0812 07:01:13.933240 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":126,"threads_started":1}
I20260812 07:01:13.935529 14754 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb11e014f9740c5a89e75cba8313dfe (DEFAULT_TABLE table=test-table [id=2cf15417dcfc401a90e3c79eeb79fe6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:01:13.935637 15216 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb11e014f9740c5a89e75cba8313dfe (DEFAULT_TABLE table=test-table [id=2cf15417dcfc401a90e3c79eeb79fe6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:01:13.936184 14987 tablet_service.cc:1467] Processing CreateTablet for tablet 5fb11e014f9740c5a89e75cba8313dfe (DEFAULT_TABLE table=test-table [id=2cf15417dcfc401a90e3c79eeb79fe6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:01:13.936640 14754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb11e014f9740c5a89e75cba8313dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:01:13.936693 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb11e014f9740c5a89e75cba8313dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:01:13.937124 14987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fb11e014f9740c5a89e75cba8313dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:01:13.943163 15358 tablet_bootstrap.cc:492] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: Bootstrap starting.
I20260812 07:01:13.944510 15361 tablet_bootstrap.cc:492] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Bootstrap starting.
I20260812 07:01:13.945331 15358 tablet_bootstrap.cc:654] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: Neither blocks nor log segments found. Creating new log.
I20260812 07:01:13.946281 15361 tablet_bootstrap.cc:654] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Neither blocks nor log segments found. Creating new log.
I20260812 07:01:13.946574 15358 log.cc:824] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: Log is configured to *not* fsync() on all Append() calls
I20260812 07:01:13.947409 15359 tablet_bootstrap.cc:492] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Bootstrap starting.
I20260812 07:01:13.947639 15361 log.cc:824] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Log is configured to *not* fsync() on all Append() calls
I20260812 07:01:13.948815 15358 tablet_bootstrap.cc:492] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: No bootstrap required, opened a new log
I20260812 07:01:13.949007 15358 ts_tablet_manager.cc:1397] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:01:13.949709 15361 tablet_bootstrap.cc:492] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: No bootstrap required, opened a new log
I20260812 07:01:13.949815 15359 tablet_bootstrap.cc:654] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Neither blocks nor log segments found. Creating new log.
I20260812 07:01:13.949889 15361 ts_tablet_manager.cc:1397] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:01:13.950515 15359 log.cc:824] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Log is configured to *not* fsync() on all Append() calls
I20260812 07:01:13.952136 15359 tablet_bootstrap.cc:492] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: No bootstrap required, opened a new log
I20260812 07:01:13.952323 15359 ts_tablet_manager.cc:1397] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260812 07:01:13.952905 15358 raft_consensus.cc:348] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:13.953192 15358 raft_consensus.cc:374] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:01:13.953284 15358 raft_consensus.cc:729] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3dc8b2d114418187f7d0d759600573, State: Initialized, Role: FOLLOWER
I20260812 07:01:13.953735 15358 consensus_queue.cc:260] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:13.953652 15361 raft_consensus.cc:348] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:13.953881 15361 raft_consensus.cc:374] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:01:13.953972 15361 raft_consensus.cc:729] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1a03a46083e4d07a6548ae5d8106358, State: Initialized, Role: FOLLOWER
I20260812 07:01:13.954455 15361 consensus_queue.cc:260] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:13.954759 15358 ts_tablet_manager.cc:1428] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:01:13.955194 14866 heartbeater.cc:503] Master 127.27.110.254:37973 was elected leader, sending a full tablet report...
I20260812 07:01:13.955188 15327 heartbeater.cc:503] Master 127.27.110.254:37973 was elected leader, sending a full tablet report...
I20260812 07:01:13.955905 15359 raft_consensus.cc:348] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:13.956226 15359 raft_consensus.cc:374] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:01:13.956328 15359 raft_consensus.cc:729] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10495a5015614b1abbb0851ddb9090fe, State: Initialized, Role: FOLLOWER
I20260812 07:01:13.956562 15361 ts_tablet_manager.cc:1428] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:01:13.956758 15359 consensus_queue.cc:260] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:13.957412 15098 heartbeater.cc:503] Master 127.27.110.254:37973 was elected leader, sending a full tablet report...
I20260812 07:01:13.957720 15359 ts_tablet_manager.cc:1428] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260812 07:01:14.125353 15328 tablet.cc:2369] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:01:14.156141 15099 tablet.cc:2369] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:01:14.179543 15374 raft_consensus.cc:482] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:01:14.179797 15374 raft_consensus.cc:504] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:14.181164 15374 leader_election.cc:288] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1a03a46083e4d07a6548ae5d8106358 (127.27.110.195:37083), 5e3dc8b2d114418187f7d0d759600573 (127.27.110.193:40195)
W20260812 07:01:14.182837 14867 tablet.cc:2369] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:01:14.185691 15357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.181632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:46411 (local address 127.27.110.195:37083)
0812 07:01:14.181795 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:14.181798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:14.181865 (+ 67us) server_negotiation.cc:407] Connection header received
0812 07:01:14.182041 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:14.182045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:14.182104 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:14.182202 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:01:14.182671 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.183507 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.184611 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.185131 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.185264 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:14.185319 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:14.185496 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 07:01:14.185567 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:01:14.185720 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.181435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:37083 (local address 127.27.110.194:46411)
0812 07:01:14.181752 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:14.181769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:14.181870 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:14.182224 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:14.182227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:14.182247 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:01:14.182372 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:14.182379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.183643 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.183646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:14.184481 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:14.184490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.185259 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.185262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:14.185359 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:14.185389 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:01:14.185478 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 07:01:14.185553 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260812 07:01:14.185899 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.182107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:40565 (local address 127.27.110.193:40195)
0812 07:01:14.182254 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:14.182259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:14.182413 (+ 154us) server_negotiation.cc:407] Connection header received
0812 07:01:14.182576 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:14.182579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:14.182649 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:14.182722 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:01:14.183020 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.183753 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.184861 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.185251 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.185396 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:14.185457 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:14.185663 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 07:01:14.185742 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:01:14.186029 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.181916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40195 (local address 127.27.110.194:40565)
0812 07:01:14.182324 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:14.182335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:01:14.182452 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:01:14.182736 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:01:14.182739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:01:14.182758 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:01:14.182874 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:14.182881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.183868 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.183870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:14.184736 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:01:14.184746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.185388 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.185392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:01:14.185520 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:14.185538 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 07:01:14.185644 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:01:14.185717 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":24,"thread_start_us":76,"threads_started":1}
I20260812 07:01:14.186755 14785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb11e014f9740c5a89e75cba8313dfe" candidate_uuid: "10495a5015614b1abbb0851ddb9090fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e3dc8b2d114418187f7d0d759600573" is_pre_election: true
I20260812 07:01:14.186755 15244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb11e014f9740c5a89e75cba8313dfe" candidate_uuid: "10495a5015614b1abbb0851ddb9090fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a03a46083e4d07a6548ae5d8106358" is_pre_election: true
I20260812 07:01:14.187099 14785 raft_consensus.cc:2454] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10495a5015614b1abbb0851ddb9090fe in term 0.
I20260812 07:01:14.187099 15244 raft_consensus.cc:2454] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10495a5015614b1abbb0851ddb9090fe in term 0.
I20260812 07:01:14.187562 14910 leader_election.cc:302] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10495a5015614b1abbb0851ddb9090fe, b1a03a46083e4d07a6548ae5d8106358; no voters:
I20260812 07:01:14.187805 15374 raft_consensus.cc:2781] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:01:14.187911 15374 raft_consensus.cc:482] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:01:14.187978 15374 raft_consensus.cc:3037] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:01:14.189043 15374 raft_consensus.cc:504] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:14.189391 15374 leader_election.cc:288] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [CANDIDATE]: Term 1 election: Requested vote from peers b1a03a46083e4d07a6548ae5d8106358 (127.27.110.195:37083), 5e3dc8b2d114418187f7d0d759600573 (127.27.110.193:40195)
I20260812 07:01:14.189826 15244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb11e014f9740c5a89e75cba8313dfe" candidate_uuid: "10495a5015614b1abbb0851ddb9090fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1a03a46083e4d07a6548ae5d8106358"
I20260812 07:01:14.189826 14785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fb11e014f9740c5a89e75cba8313dfe" candidate_uuid: "10495a5015614b1abbb0851ddb9090fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e3dc8b2d114418187f7d0d759600573"
I20260812 07:01:14.189954 14785 raft_consensus.cc:3037] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:01:14.189954 15244 raft_consensus.cc:3037] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:01:14.191339 14785 raft_consensus.cc:2454] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10495a5015614b1abbb0851ddb9090fe in term 1.
I20260812 07:01:14.191339 15244 raft_consensus.cc:2454] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10495a5015614b1abbb0851ddb9090fe in term 1.
I20260812 07:01:14.191628 14910 leader_election.cc:302] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10495a5015614b1abbb0851ddb9090fe, 5e3dc8b2d114418187f7d0d759600573; no voters:
I20260812 07:01:14.191815 15374 raft_consensus.cc:2781] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:01:14.192077 15374 raft_consensus.cc:686] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 1 LEADER]: Becoming Leader. State: Replica: 10495a5015614b1abbb0851ddb9090fe, State: Running, Role: LEADER
I20260812 07:01:14.192425 15374 consensus_queue.cc:237] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } }
I20260812 07:01:14.195706 14524 catalog_manager.cc:5495] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 10495a5015614b1abbb0851ddb9090fe (127.27.110.194). New cstate: current_term: 1 leader_uuid: "10495a5015614b1abbb0851ddb9090fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10495a5015614b1abbb0851ddb9090fe" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 } health_report { overall_health: UNKNOWN } } }
I20260812 07:01:14.222859 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.219197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.27.110.194:41815)
0812 07:01:14.219331 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:01:14.219335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:01:14.219349 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:01:14.219428 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:01:14.219430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:01:14.219485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:01:14.219597 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:01:14.219825 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.220347 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.221240 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:01:14.221452 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:01:14.221533 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:01:14.221974 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:01:14.222440 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:01:14.222588 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:01:14.222693 (+ 105us) server_negotiation.cc:299] Negotiation successful
0812 07:01:14.222749 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:01:14.229128 15244 raft_consensus.cc:1261] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 1 FOLLOWER]: Refusing update from remote peer 10495a5015614b1abbb0851ddb9090fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:01:14.229138 14785 raft_consensus.cc:1261] T 5fb11e014f9740c5a89e75cba8313dfe P 5e3dc8b2d114418187f7d0d759600573 [term 1 FOLLOWER]: Refusing update from remote peer 10495a5015614b1abbb0851ddb9090fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:01:14.229890 15378 consensus_queue.cc:1035] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1a03a46083e4d07a6548ae5d8106358" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:01:14.230172 15374 consensus_queue.cc:1035] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e3dc8b2d114418187f7d0d759600573" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:01:14.239089 15383 mvcc.cc:205] Tried to move back new op lower bound from 7317578032034938880 to 7317578031895224320. Current Snapshot: MvccSnapshot[applied={T|T < 7317578032034938880}]
I20260812 07:01:14.246807 14524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:01:14.246959 14524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:01:14.249351 14524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 291ca746fd5b4a5fb9d671d8138a0240: Sending DeleteTablet for 3 replicas of tablet 5fb11e014f9740c5a89e75cba8313dfe
I20260812 07:01:14.250089 14987 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb11e014f9740c5a89e75cba8313dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:01:14 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43454
I20260812 07:01:14.250237 14754 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb11e014f9740c5a89e75cba8313dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:01:14 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:39956
I20260812 07:01:14.250267 15216 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fb11e014f9740c5a89e75cba8313dfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:01:14 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:58100
I20260812 07:01:14.250762 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 14641
I20260812 07:01:14.263378 15399 tablet_replica.cc:330] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: stopping tablet replica
I20260812 07:01:14.263827 15399 raft_consensus.cc:2229] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:01:14.264376 15399 raft_consensus.cc:2258] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:01:14.264833 15401 tablet_replica.cc:330] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: stopping tablet replica
I20260812 07:01:14.265230 15401 raft_consensus.cc:2229] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:01:14.265616 15401 raft_consensus.cc:2258] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:01:14.266119 15399 ts_tablet_manager.cc:1905] T 5fb11e014f9740c5a89e75cba8313dfe P 10495a5015614b1abbb0851ddb9090fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:01:14.267210 15401 ts_tablet_manager.cc:1905] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 07:01:14.268937 14501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:40195 (error 108)
I20260812 07:01:14.269238 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 14872
I20260812 07:01:14.282325 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.281848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40195 (local address 127.0.0.1:39970)
0812 07:01:14.282033 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:14.282176 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 07:01:14.282819 14501 catalog_manager.cc:4584] TS 5e3dc8b2d114418187f7d0d759600573 (127.27.110.193:40195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5fb11e014f9740c5a89e75cba8313dfe: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40195: connect: Connection refused (error 111)
I20260812 07:01:14.283218 15401 ts_tablet_manager.cc:1918] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:01:14.283396 15401 log.cc:1196] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/5fb11e014f9740c5a89e75cba8313dfe
I20260812 07:01:14.283766 15401 ts_tablet_manager.cc:1939] T 5fb11e014f9740c5a89e75cba8313dfe P b1a03a46083e4d07a6548ae5d8106358: Deleting consensus metadata
I20260812 07:01:14.285161 14501 catalog_manager.cc:4841] TS b1a03a46083e4d07a6548ae5d8106358 (127.27.110.195:37083): tablet 5fb11e014f9740c5a89e75cba8313dfe (table test-table [id=2cf15417dcfc401a90e3c79eeb79fe6d]) successfully deleted
I20260812 07:01:14.289479 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 15105
I20260812 07:01:14.294127 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:01:14.289683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:41815 (local address 127.0.0.1:43476)
0812 07:01:14.293930 (+ 4247us) negotiation.cc:107] Waiting for socket to connect
0812 07:01:14.294011 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.194:41815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4121}
I20260812 07:01:14.309019 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 14463
2026-08-12T07:01:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2771 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:01:14 dist-test-slave-w4v5 krb5kdc[15408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:01:14 dist-test-slave-w4v5 krb5kdc[15408](info): set up 2 sockets
Aug 12 07:01:14 dist-test-slave-w4v5 krb5kdc[15408](info): commencing operation
krb5kdc: starting...
W20260812 07:01:15.625379 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:01:15 dist-test-slave-w4v5 krb5kdc[15408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:01:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/rangeradmin_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/rangeradmin_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/rangerlookup_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/rangerlookup_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
I20260812 07:01:15.805471 28091 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh57nMD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/postgres -l logfile start
2026-08-12 07:01:18.365 UTC [15462] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 07:01:18.365 UTC [15462] LOG: listening on IPv4 address "127.27.110.212", port 47905
2026-08-12 07:01:18.369 UTC [15462] LOG: listening on Unix socket "/tmp/.s.PGSQL.47905"
2026-08-12 07:01:18.392 UTC [15468] LOG: database system was shut down at 2026-08-12 07:01:16 UTC
2026-08-12 07:01:18.396 UTC [15462] LOG: database system is ready to accept connections
I20260812 07:01:19.611759 28091 mini_postgres.cc:96] Postgres bound to 47905
2026-08-12 07:01:19.621 UTC [15479] FATAL: database "slave" does not exist
127.27.110.212:47905 - accepting connections
I20260812 07:01:19.622282 28091 mini_ranger.cc:161] Starting Ranger...
I20260812 07:01:19.640429 28091 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 07:01:20.142298 28091 mini_ranger.cc:87] Created ranger Postgres database
I20260812 07:01:20.142408 28091 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-admin
2026-08-12 07:01:20,490 [I] DB FLAVOR :POSTGRES
2026-08-12 07:01:20,490 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:01:20,491 [I] Checking connection..
2026-08-12 07:01:20,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 07:01:20,745 [I] Checking connection passed.
2026-08-12 07:01:20,745 [I] --------- Verifying version history table ---------
2026-08-12 07:01:20,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:01:21,020 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:01:21,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:01:21,285 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:01:21,285 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 07:01:21,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 07:01:21.555 UTC [15603] WARNING: there is no transaction in progress
2026-08-12 07:01:21,563 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 07:01:21,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:01:21,839 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 07:01:21,839 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:01:21,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 07:01:22,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 07:01:22,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w4v5.c.gcp-upstream.internal','N') ;"
2026-08-12 07:01:22,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:01:22,892 [I] Table x_portal_user does not exist in database ranger
2026-08-12 07:01:22,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:01:23,149 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 07:01:23,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:01:23,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:01:23,667 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 07:01:23,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 07:01:24.642 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.658 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.675 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.691 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.707 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.726 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.786 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.793 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.807 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.818 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.829 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.842 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.851 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.857 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.863 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:24.869 UTC [15806] WARNING: there is no transaction in progress
2026-08-12 07:01:25,499 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 07:01:25,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 07:01:25,748 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 07:01:25,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:01:25,991 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 07:01:25,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:01:26,262 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 07:01:26,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:01:26,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:01:26,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w4v5.c.gcp-upstream.internal';"
2026-08-12 07:01:27,034 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 07:01:27,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 07:01:27,299 [I] DB_PATCHES have already been applied
I20260812 07:01:27.305282 28091 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh57nMD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 07:01:27.305346 28091 mini_ranger.cc:193] Using host: 127.27.110.212
I20260812 07:01:27.310007 28091 mini_ranger.cc:227] Ranger admin URL: http://127.27.110.212:33755
Aug 12, 2026 7:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 7:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 7:01:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 7:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 7:01:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 07:01:28.706182 28091 mini_ranger.cc:160] Time spent starting Ranger: real 9.084s user 0.000s sys 0.014s
Aug 12, 2026 7:01:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 7:01:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 12, 2026 7:01:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 7:01:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 7:01:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 07:01:51.701161 28091 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/rangerkms_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/rangerkms_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.110.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.110.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 07:01:51 dist-test-slave-w4v5 krb5kdc[15408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518111, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 07:01:51.794778 28091 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 07:01:51.806331 28091 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 07:01:52.621274 28091 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 07:01:52.621384 28091 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms
I20260812 07:01:52.621490 28091 mini_ranger_kms.cc:227] Using postgres at 127.27.110.212:47905
2026-08-12 07:01:52,927 [I] DB FLAVOR :POSTGRES
2026-08-12 07:01:52,928 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:01:52,928 [I] Checking connection
2026-08-12 07:01:52,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 07:01:53,164 [I] connection success
2026-08-12 07:01:53,165 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 07:01:53,165 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 07:01:53,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 07:01:53,430 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 07:01:53,431 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:01:53,431 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 07:01:53,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:47905/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 07:01:53,687 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 07:01:54.185432 28091 mini_ranger_kms.cc:310] Created kms service
I20260812 07:01:54.433629 28091 mini_ranger_kms.cc:326] Created kudu user
I20260812 07:01:54.481707 28091 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 07:01:54 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518111, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:01:54.902743 28091 mini_ranger_kms.cc:384] Added ranger policy
I20260812 07:01:54.902896 28091 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 07:01:54.902915 28091 mini_ranger_kms.cc:240] Using host: 127.27.110.212
I20260812 07:01:54.907843 28091 mini_ranger_kms.cc:276] Ranger KMS PID: 16159
I20260812 07:01:54.907984 28091 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.110.212:58373
Aug 12, 2026 7:01:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:01:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:01:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 7:01:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 7:01:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 7:01:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 07:01:57 dist-test-slave-w4v5 krb5kdc[15408](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 07:01:57 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:01:58 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:02:28 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:02:28 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:02:58 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:02:58 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:03:28 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:03:28 dist-test-slave-w4v5 krb5kdc[15408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518117, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:03:56.082428 28091 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 124.288s user 0.041s sys 0.417s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger KMS service: process with PID 16159 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p16159
f196
n127.0.0.1:45860->127.27.110.212:47905'
I20260812 07:03:56.083596 28091 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 07:03:56.114400 28091 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 07:03:56.114557 28091 mini_ranger.cc:66] Stopping Ranger...
I20260812 07:03:56.479856 28091 mini_ranger.cc:68] Stopped Ranger
2026-08-12 07:03:56.480 UTC [15462] LOG: received smart shutdown request
2026-08-12 07:03:56.483 UTC [15462] LOG: background worker "logical replication launcher" (PID 15477) exited with exit code 1
2026-08-12 07:03:56.484 UTC [15469] LOG: shutting down
2026-08-12 07:03:56.614 UTC [15462] LOG: database system is shut down
2026-08-12T07:03:56Z chronyd exiting
I20260812 07:03:56.633971 28091 test_util.cc:175] -----------------------------------------------
I20260812 07:03:56.634034 28091 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786518006756923-28091-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (162295 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:03:56 dist-test-slave-w4v5 krb5kdc[16376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:03:56 dist-test-slave-w4v5 krb5kdc[16376](info): set up 2 sockets
Aug 12 07:03:56 dist-test-slave-w4v5 krb5kdc[16376](info): commencing operation
krb5kdc: starting...
W20260812 07:03:57.916987 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:03:58 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:03:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:03:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/rangeradmin_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/rangeradmin_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/rangerlookup_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/rangerlookup_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
I20260812 07:03:58.101469 28091 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskh57nMD/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/postgres -l logfile start
2026-08-12 07:04:00.563 UTC [16432] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 07:04:00.563 UTC [16432] LOG: listening on IPv4 address "127.27.110.212", port 44291
2026-08-12 07:04:00.566 UTC [16432] LOG: listening on Unix socket "/tmp/.s.PGSQL.44291"
2026-08-12 07:04:00.588 UTC [16441] LOG: database system was shut down at 2026-08-12 07:03:59 UTC
2026-08-12 07:04:00.592 UTC [16432] LOG: database system is ready to accept connections
I20260812 07:04:01.810468 28091 mini_postgres.cc:96] Postgres bound to 44291
2026-08-12 07:04:01.819 UTC [16456] FATAL: database "slave" does not exist
127.27.110.212:44291 - accepting connections
I20260812 07:04:01.820397 28091 mini_ranger.cc:161] Starting Ranger...
I20260812 07:04:01.838482 28091 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 07:04:02.323931 28091 mini_ranger.cc:87] Created ranger Postgres database
I20260812 07:04:02.324024 28091 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-admin
2026-08-12 07:04:02,622 [I] DB FLAVOR :POSTGRES
2026-08-12 07:04:02,623 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:04:02,623 [I] Checking connection..
2026-08-12 07:04:02,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 07:04:02,897 [I] Checking connection passed.
2026-08-12 07:04:02,897 [I] --------- Verifying version history table ---------
2026-08-12 07:04:02,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:04:03,141 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:04:03,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:04:03,382 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 07:04:03,382 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 07:04:03,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 07:04:03.639 UTC [16569] WARNING: there is no transaction in progress
2026-08-12 07:04:03,646 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 07:04:03,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 07:04:03,918 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 07:04:03,919 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:04:03,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 07:04:04,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 07:04:04,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-w4v5.c.gcp-upstream.internal','N') ;"
2026-08-12 07:04:04,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:04:04,954 [I] Table x_portal_user does not exist in database ranger
2026-08-12 07:04:04,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:04:05,209 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 07:04:05,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:04:05,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:04:05,734 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 07:04:05,735 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 07:04:06.661 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.676 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.691 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.706 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.723 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.740 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.795 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.802 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.816 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.826 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.836 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.850 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.859 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.865 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.871 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:06.877 UTC [16751] WARNING: there is no transaction in progress
2026-08-12 07:04:07,492 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 07:04:07,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 07:04:07,748 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 07:04:07,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 07:04:08,017 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 07:04:08,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 07:04:08,274 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 07:04:08,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 07:04:08,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 07:04:08,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-w4v5.c.gcp-upstream.internal';"
2026-08-12 07:04:09,020 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 07:04:09,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 07:04:09,277 [I] DB_PATCHES have already been applied
I20260812 07:04:09.283255 28091 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-admin:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskh57nMD/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 07:04:09.283346 28091 mini_ranger.cc:193] Using host: 127.27.110.212
I20260812 07:04:09.288376 28091 mini_ranger.cc:227] Ranger admin URL: http://127.27.110.212:54673
Aug 12, 2026 7:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 7:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 7:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 7:04:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 7:04:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 07:04:10.716924 28091 mini_ranger.cc:160] Time spent starting Ranger: real 8.897s user 0.000s sys 0.018s
Aug 12, 2026 7:04:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 7:04:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 7:04:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 7:04:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 7:04:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 07:04:33.429486 28091 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/rangerkms_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/rangerkms_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.110.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.110.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.110.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/HTTP_127.27.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 07:04:33 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 07:04:33.515823 28091 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 07:04:33.526355 28091 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 07:04:34.379559 28091 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 07:04:34.379653 28091 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms
I20260812 07:04:34.379673 28091 mini_ranger_kms.cc:227] Using postgres at 127.27.110.212:44291
2026-08-12 07:04:34,590 [I] DB FLAVOR :POSTGRES
2026-08-12 07:04:34,590 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 07:04:34,590 [I] Checking connection
2026-08-12 07:04:34,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 07:04:34,834 [I] connection success
2026-08-12 07:04:34,834 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 07:04:34,834 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 07:04:34,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 07:04:35,073 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 07:04:35,073 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 07:04:35,074 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 07:04:35,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.110.212:44291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 07:04:35,333 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 07:04:35.804862 28091 mini_ranger_kms.cc:310] Created kms service
I20260812 07:04:36.013547 28091 mini_ranger_kms.cc:326] Created kudu user
I20260812 07:04:36.059813 28091 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 07:04:36 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518273, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:36.507213 28091 mini_ranger_kms.cc:384] Added ranger policy
I20260812 07:04:36.507367 28091 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms:/tmp/dist-test-taskh57nMD/build/debug/bin/postgresql.jar:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskh57nMD/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 07:04:36.507390 28091 mini_ranger_kms.cc:240] Using host: 127.27.110.212
I20260812 07:04:36.513727 28091 mini_ranger_kms.cc:276] Ranger KMS PID: 17079
I20260812 07:04:36.513819 28091 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.110.212:47439
Aug 12, 2026 7:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 7:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 7:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 7:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 7:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 7:04:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 07:04:38 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 07:04:39 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:39.386768 28091 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.871s user 0.011s sys 0.182s
I20260812 07:04:39.386906 28091 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 07:04:39.387020 28091 mini_ranger_kms.cc:401] 127.27.110.212:47439/kms/v1/keys
Aug 12 07:04:39 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518278, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.110.212@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
Aug 12 07:04:41 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:04:41.938422 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36091
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36091
--ranger_config_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:42.065752 17162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:42.066030 17162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:42.066072 17162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:42.070036 17162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:42.070099 17162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:42.070118 17162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:42.070135 17162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:42.070150 17162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:42.075739 17162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.110.212:47439/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36091
--ranger_config_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:42.077009 17162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:42.078045 17162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:04:42.087783 17162 server_base.cc:1034] running on GCE node
W20260812 07:04:42.087730 17177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:42.087800 17174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:42.087725 17173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:42.088719 17162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:42.089874 17162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:42.091066 17162 hybrid_clock.cc:648] HybridClock initialized: now 1786518282091045 us; error 33 us; skew 500 ppm
Aug 12 07:04:42 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:42.094228 17162 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:42.095244 17162 webserver.cc:460] Webserver started at http://127.27.110.254:35153/ using document root <none> and password file <none>
I20260812 07:04:42.095811 17162 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:42.095870 17162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:42.096098 17162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:04:42 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:42.159145 17162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "9ace375dbb4d4f569a3aef22fdd11598"
format_stamp: "Formatted at 2026-08-12 07:04:42 on dist-test-slave-w4v5"
server_key: "105434497b655f5aa151d76f545ab073"
server_key_iv: "deeab9ae14876dafb5ab9d218518d26b"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:42.159996 17162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "9ace375dbb4d4f569a3aef22fdd11598"
format_stamp: "Formatted at 2026-08-12 07:04:42 on dist-test-slave-w4v5"
server_key: "105434497b655f5aa151d76f545ab073"
server_key_iv: "deeab9ae14876dafb5ab9d218518d26b"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:42.165298 17162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:42.185606 17185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:42.187288 17162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:42.187429 17162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "9ace375dbb4d4f569a3aef22fdd11598"
format_stamp: "Formatted at 2026-08-12 07:04:42 on dist-test-slave-w4v5"
server_key: "105434497b655f5aa151d76f545ab073"
server_key_iv: "deeab9ae14876dafb5ab9d218518d26b"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:42.187546 17162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:42.200804 17162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:42.203568 17162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:42.203735 17162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:42.211611 17162 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36091
I20260812 07:04:42.211624 17275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36091 every 8 connection(s)
I20260812 07:04:42.212623 17162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:42.215653 17276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:42.218670 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 17162
I20260812 07:04:42.218832 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:04:42.219211 28091 external_mini_cluster.cc:1428] Setting key 3a7e1e63514f75708b7bfd457e709a59
I20260812 07:04:42.222203 17276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Bootstrap starting.
I20260812 07:04:42.224607 17276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:42.225405 17276 log.cc:824] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:42.227499 17276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: No bootstrap required, opened a new log
Aug 12 07:04:42 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:42.231510 17276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:42.231832 17276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:42.231945 17276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ace375dbb4d4f569a3aef22fdd11598, State: Initialized, Role: FOLLOWER
I20260812 07:04:42.232458 17276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:42.232594 17276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:42.232685 17276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:42.232792 17276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:42.233912 17276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:42.234324 17276 leader_election.cc:302] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ace375dbb4d4f569a3aef22fdd11598; no voters:
I20260812 07:04:42.234575 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:42.220865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.27.110.254:36091)
0812 07:04:42.221350 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:42.221366 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:42.221403 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:04:42.222036 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:42.222053 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:42.222462 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:42.222813 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:42.223661 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:42.224679 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:42.225579 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:42.226064 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:42.226248 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:42.229322 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:42.229338 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:42.229356 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:42.229384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:42.231254 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:42.231781 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:42.231786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:42.231790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:42.231983 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:42.232309 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:42.232313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:42.232315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:42.232889 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:42.233138 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:42.233377 (+ 239us) server_negotiation.cc:299] Negotiation successful
0812 07:04:42.233684 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":157,"threads_started":1}
I20260812 07:04:42.234741 17276 leader_election.cc:288] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:42.234997 17286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:42.235280 17286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 LEADER]: Becoming Leader. State: Replica: 9ace375dbb4d4f569a3aef22fdd11598, State: Running, Role: LEADER
I20260812 07:04:42.235589 17286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:42.236191 17276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:42.237699 17292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ace375dbb4d4f569a3aef22fdd11598. Latest consensus state: current_term: 1 leader_uuid: "9ace375dbb4d4f569a3aef22fdd11598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } } }
I20260812 07:04:42.237888 17292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:42.238346 17300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:42.238302 17291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ace375dbb4d4f569a3aef22fdd11598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } } }
I20260812 07:04:42.238926 17276 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 07:04:42.238999 17291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:42.245175 17300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:42.252733 17300 catalog_manager.cc:1347] Generated new cluster ID: 5c91ba23b2fd4a9c93efccd763c124f3
I20260812 07:04:42.252803 17300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:42.260667 17300 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:42.261721 17300 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:42.268764 17300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Generated new TSK 0
I20260812 07:04:42.269476 17300 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:04:42.996011 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36091
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:43.119956 17354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:43.120234 17354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:43.120347 17354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:43.125667 17354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:43.125746 17354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:43.125844 17354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:04:43.131304 17354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.110.212:47439/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:43.132694 17354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:43.133741 17354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:43.144680 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.144695 17370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.145076 17369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:43.145776 17354 server_base.cc:1034] running on GCE node
I20260812 07:04:43.146283 17354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:43.147058 17354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:43.148263 17354 hybrid_clock.cc:648] HybridClock initialized: now 1786518283148200 us; error 80 us; skew 500 ppm
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:43.151558 17354 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:04:43.152522 17354 webserver.cc:460] Webserver started at http://127.27.110.193:43203/ using document root <none> and password file <none>
I20260812 07:04:43.153079 17354 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:43.153190 17354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:43.153481 17354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:43.169472 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "7de11838adb1ae9eaab01afd52243db7"
server_key_iv: "f504f7b718bf7b7b82bc55dba79dfbb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.170243 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "7de11838adb1ae9eaab01afd52243db7"
server_key_iv: "f504f7b718bf7b7b82bc55dba79dfbb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.177891 17354 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260812 07:04:43.199405 17392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.200891 17354 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260812 07:04:43.201035 17354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "7de11838adb1ae9eaab01afd52243db7"
server_key_iv: "f504f7b718bf7b7b82bc55dba79dfbb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.201184 17354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:43.222316 17354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:43.225910 17354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:43.226120 17354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:43.226881 17354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:43.227994 17354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:43.228067 17354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.228144 17354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:43.228174 17354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.239801 17354 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:35439
I20260812 07:04:43.241029 17354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:43.246447 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 17354
I20260812 07:04:43.246568 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:43.246893 28091 external_mini_cluster.cc:1428] Setting key 57cb3212879b84b4809a30d7780e179d
I20260812 07:04:43.249873 17571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:35439 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:04:43.277812 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.252334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36091 (local address 127.27.110.193:41467)
0812 07:04:43.254840 (+ 2506us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:43.254918 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:43.255966 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:43.256904 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:43.256913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:43.257369 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:43.258063 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:43.258084 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.259326 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.259332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:43.260897 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:43.260912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.261851 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.261858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:43.262102 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:43.262934 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:43.262962 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:43.269234 (+ 6272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:43.274037 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:43.274045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:43.274066 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:43.274463 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:43.274816 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:43.274819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:43.274821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:43.275027 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:43.275651 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:43.275660 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:43.276012 (+ 352us) client_negotiation.cc:769] Sending connection context
0812 07:04:43.276229 (+ 217us) client_negotiation.cc:240] Negotiation successful
0812 07:04:43.276624 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2205,"thread_start_us":149,"threads_started":1}
I20260812 07:04:43.278631 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.253480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:41467 (local address 127.27.110.254:36091)
0812 07:04:43.255132 (+ 1652us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:43.255141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:43.255158 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:04:43.256349 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:43.256353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:43.256413 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:43.256557 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:43.258249 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.259168 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.261068 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.261504 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.261634 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:43.269493 (+ 7859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:43.269498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:43.269501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:43.269534 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:43.273868 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:43.274594 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:43.274598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:43.274600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:43.274701 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:43.275206 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:43.275210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:43.275212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:43.275493 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:43.275592 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:43.278312 (+ 2720us) server_negotiation.cc:299] Negotiation successful
0812 07:04:43.278456 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1540,"thread_start_us":118,"threads_started":1}
I20260812 07:04:43.280606 17573 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36091
I20260812 07:04:43.280864 17573 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:43.281458 17573 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:43.283331 17213 ts_manager.cc:194] Registered new tserver with Master: dd35df0700b5429fa56a2e5d3a2d0e7e (127.27.110.193:35439)
I20260812 07:04:43.285127 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:41467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:04:43.306481 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36091
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 07:04:43.455423 17591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:43.455792 17591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:43.455859 17591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:43.461803 17591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:43.461882 17591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:43.461978 17591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:04:43.469897 17591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.110.212:47439/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:43.471382 17591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:43.472781 17591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:43.486222 17601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.486513 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.486703 17603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:43.486984 17591 server_base.cc:1034] running on GCE node
I20260812 07:04:43.487494 17591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:43.488307 17591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:43.489526 17591 hybrid_clock.cc:648] HybridClock initialized: now 1786518283489458 us; error 83 us; skew 500 ppm
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:43.493342 17591 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:04:43.494478 17591 webserver.cc:460] Webserver started at http://127.27.110.194:37045/ using document root <none> and password file <none>
I20260812 07:04:43.495388 17591 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:43.495502 17591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:43.495796 17591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:43.514274 17591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "22220d415e0341c19e9bd8cd333b1962"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "e2f90102b845490105ce2e98871f9972"
server_key_iv: "771cac7c09ba70c02ef68b9981d3f8fe"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.515252 17591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "22220d415e0341c19e9bd8cd333b1962"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "e2f90102b845490105ce2e98871f9972"
server_key_iv: "771cac7c09ba70c02ef68b9981d3f8fe"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.521072 17591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:43.541807 17610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.543514 17591 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 07:04:43.543663 17591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "22220d415e0341c19e9bd8cd333b1962"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "e2f90102b845490105ce2e98871f9972"
server_key_iv: "771cac7c09ba70c02ef68b9981d3f8fe"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.543836 17591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:43.561311 17591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:43.565248 17591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:43.565450 17591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:43.566136 17591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:43.567214 17591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:43.567282 17591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.567348 17591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:43.567379 17591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.579396 17591 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:34231
I20260812 07:04:43.579547 17814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:34231 every 8 connection(s)
I20260812 07:04:43.580444 17591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:43.588137 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 17591
I20260812 07:04:43.588269 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:43.588574 28091 external_mini_cluster.cc:1428] Setting key c8d32b28926f632b2fe404b2ad35b358
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:43.597679 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.582365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:40689 (local address 127.27.110.254:36091)
0812 07:04:43.582503 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:43.582507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:43.585321 (+ 2814us) server_negotiation.cc:407] Connection header received
0812 07:04:43.586515 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:43.586519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:43.586587 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:43.586690 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:43.588158 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.588838 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.590244 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.590627 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.590729 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:43.593798 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:43.593805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:43.593811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:43.593850 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:43.595537 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:43.596112 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:43.596116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:43.596118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:43.596201 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:43.596559 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:43.596562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:43.596563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:43.596754 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:43.596866 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:43.597371 (+ 505us) server_negotiation.cc:299] Negotiation successful
0812 07:04:43.597503 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:04:43.598590 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.582906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36091 (local address 127.27.110.194:40689)
0812 07:04:43.585120 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:43.585179 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:43.586259 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:43.586915 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:43.586924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:43.587317 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:43.587986 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:43.588002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.588976 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.588980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:43.590114 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:43.590127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:43.590794 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:43.590801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:43.591030 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:43.591752 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:43.591780 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:43.593598 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:43.595680 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:43.595686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:43.595702 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:43.595989 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:43.596296 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:43.596300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:43.596302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:43.596449 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:43.596866 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:43.596874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:43.597124 (+ 250us) client_negotiation.cc:769] Sending connection context
0812 07:04:43.597312 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 07:04:43.597659 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1981,"thread_start_us":399,"threads_started":1}
I20260812 07:04:43.599838 17816 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36091
I20260812 07:04:43.600090 17816 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:43.600601 17816 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:43.601652 17213 ts_manager.cc:194] Registered new tserver with Master: 22220d415e0341c19e9bd8cd333b1962 (127.27.110.194:34231)
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:04:43.602278 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:40689
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:04:43.629917 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36091
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
W20260812 07:04:43.799137 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:43.799515 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:43.799615 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:43.805197 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:43.805286 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:43.805397 17823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:04:43.813140 17823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.110.212:47439/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:43.814586 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:43.816076 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:43.830875 17835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.831362 17832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:43.831598 17831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:43.832769 17823 server_base.cc:1034] running on GCE node
I20260812 07:04:43.833276 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:43.834089 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:43.837349 17823 hybrid_clock.cc:648] HybridClock initialized: now 1786518283837271 us; error 85 us; skew 500 ppm
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:43.841250 17823 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:04:43.842398 17823 webserver.cc:460] Webserver started at http://127.27.110.195:43359/ using document root <none> and password file <none>
I20260812 07:04:43.843066 17823 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:43.843175 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:43.843420 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 07:04:43 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:43.863782 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "1cfa703c0cef407588f22b842b9478c8"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "07a87080bccedb6fa28e792c28f3c6a3"
server_key_iv: "f4cd07775eab82fed4d9bee0c22f0cf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.865602 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "1cfa703c0cef407588f22b842b9478c8"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "07a87080bccedb6fa28e792c28f3c6a3"
server_key_iv: "f4cd07775eab82fed4d9bee0c22f0cf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.870960 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 07:04:43.907506 17848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.909631 17823 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260812 07:04:43.909811 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "1cfa703c0cef407588f22b842b9478c8"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "07a87080bccedb6fa28e792c28f3c6a3"
server_key_iv: "f4cd07775eab82fed4d9bee0c22f0cf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:43.909982 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:43.953154 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:43.969195 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:43.969414 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:43.970172 17823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:43.971305 17823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:43.971369 17823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.971436 17823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:43.971467 17823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:43.982568 17823 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:33185
I20260812 07:04:43.983789 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:43.992808 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 17823
I20260812 07:04:43.992938 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:43.993242 28091 external_mini_cluster.cc:1428] Setting key 2d825aaa96e4f14588a4530602d9ec89
I20260812 07:04:43.992406 18036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:33185 every 8 connection(s)
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:44.014027 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.998146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:59489 (local address 127.27.110.254:36091)
0812 07:04:43.998275 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:43.998280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:44.000196 (+ 1916us) server_negotiation.cc:407] Connection header received
0812 07:04:44.001466 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:44.001469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:44.001523 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:44.001590 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:44.003460 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.004311 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.006042 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.006353 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.006427 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.009671 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:44.009675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:44.009678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:44.009705 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:44.011572 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.012198 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.012202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.012204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.012289 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.012695 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.012698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.012700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.012917 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:44.012996 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:44.013684 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 07:04:44.013840 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:04:44.015767 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:43.998739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36091 (local address 127.27.110.195:59489)
0812 07:04:44.000030 (+ 1291us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:44.000093 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:44.001213 (+ 1120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:44.002014 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:44.002024 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:44.002529 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:44.003282 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.003304 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.004479 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.004482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.005896 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.005907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.006658 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.006663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.006875 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.007689 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:44.007710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:44.009459 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:44.011719 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.011725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.011744 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.012082 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.012392 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.012395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.012397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.012585 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.013064 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:44.013071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:44.013387 (+ 316us) client_negotiation.cc:769] Sending connection context
0812 07:04:44.014284 (+ 897us) client_negotiation.cc:240] Negotiation successful
0812 07:04:44.014650 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":198,"threads_started":1}
I20260812 07:04:44.017211 18037 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36091
I20260812 07:04:44.017485 18037 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.018138 18037 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:44.019251 17212 ts_manager.cc:194] Registered new tserver with Master: 1cfa703c0cef407588f22b842b9478c8 (127.27.110.195:33185)
I20260812 07:04:44.020059 17212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:59489
I20260812 07:04:44.031225 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:04:44.031855 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 17354
I20260812 07:04:44.072902 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 17591
I20260812 07:04:44.088253 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 17823
I20260812 07:04:44.109472 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 17162
I20260812 07:04:44.128466 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36091
--webserver_interface=127.27.110.254
--webserver_port=35153
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36091
--ranger_config_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:44.241569 18043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:44.241850 18043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:44.241888 18043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:44.245959 18043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:44.246027 18043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:44.246047 18043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:44.246062 18043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:44.246078 18043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:44.252044 18043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36091
--ranger_config_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=35153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:44.253167 18043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:44.254259 18043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:04:44.263372 18043 server_base.cc:1034] running on GCE node
W20260812 07:04:44.263335 18059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.263321 18053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.263309 18054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:44.264312 18043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:44.265476 18043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:44.266685 18043 hybrid_clock.cc:648] HybridClock initialized: now 1786518284266648 us; error 49 us; skew 500 ppm
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:44.269640 18043 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:44.270601 18043 webserver.cc:460] Webserver started at http://127.27.110.254:35153/ using document root <none> and password file <none>
I20260812 07:04:44.271186 18043 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:44.271265 18043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:44.356349 18043 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:04:44.358948 18070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.360369 18043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:44.360536 18043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "9ace375dbb4d4f569a3aef22fdd11598"
format_stamp: "Formatted at 2026-08-12 07:04:42 on dist-test-slave-w4v5"
server_key: "105434497b655f5aa151d76f545ab073"
server_key_iv: "deeab9ae14876dafb5ab9d218518d26b"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:44.361167 18043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:44.374601 18043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:44.377357 18043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:44.377568 18043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:44.384946 18043 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36091
I20260812 07:04:44.384955 18153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36091 every 8 connection(s)
I20260812 07:04:44.386134 18043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:44.387256 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 18043
I20260812 07:04:44.387825 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:35439
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:36091
--webserver_port=43203
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:44.391508 18154 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:04:44.393220 18154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Bootstrap starting.
I20260812 07:04:44.409948 18154 log.cc:824] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:44.416728 18154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:44.417179 18154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Bootstrap complete.
I20260812 07:04:44.421465 18154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:44.421845 18154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ace375dbb4d4f569a3aef22fdd11598, State: Initialized, Role: FOLLOWER
I20260812 07:04:44.422391 18154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:44.422508 18154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:44.422598 18154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:44.422698 18154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:04:44.424681 18154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:44.425024 18154 leader_election.cc:302] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ace375dbb4d4f569a3aef22fdd11598; no voters:
I20260812 07:04:44.425364 18154 leader_election.cc:288] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:04:44.425454 18160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:04:44.425670 18160 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [term 2 LEADER]: Becoming Leader. State: Replica: 9ace375dbb4d4f569a3aef22fdd11598, State: Running, Role: LEADER
I20260812 07:04:44.425961 18160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } }
I20260812 07:04:44.426605 18154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:44.427351 18161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9ace375dbb4d4f569a3aef22fdd11598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } } }
I20260812 07:04:44.427493 18161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:44.428090 18173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:44.428120 18163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ace375dbb4d4f569a3aef22fdd11598. Latest consensus state: current_term: 2 leader_uuid: "9ace375dbb4d4f569a3aef22fdd11598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ace375dbb4d4f569a3aef22fdd11598" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36091 } } }
I20260812 07:04:44.428241 18163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598 [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:44.431559 18173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:44.432081 18154 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 07:04:44.432308 18173 catalog_manager.cc:1259] Loaded cluster ID: 5c91ba23b2fd4a9c93efccd763c124f3
I20260812 07:04:44.432404 18173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:44.434876 18173 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:44.435684 18173 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Loaded TSK: 0
I20260812 07:04:44.437125 18173 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:04:44.527187 18157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:44.527472 18157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:44.527552 18157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:44.532387 18157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:44.532472 18157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:44.532577 18157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:04:44.538383 18157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:35439
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=43203
--tserver_master_addrs=127.27.110.254:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:44.539549 18157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:44.540573 18157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:44.552981 18222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.553074 18217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.553591 18219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:44.554260 18157 server_base.cc:1034] running on GCE node
I20260812 07:04:44.554701 18157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:44.555442 18157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:44.556645 18157 hybrid_clock.cc:648] HybridClock initialized: now 1786518284556611 us; error 46 us; skew 500 ppm
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:44.563139 18157 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:04:44.564040 18157 webserver.cc:460] Webserver started at http://127.27.110.193:43203/ using document root <none> and password file <none>
I20260812 07:04:44.564827 18157 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:44.564963 18157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:44.587935 18157 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260812 07:04:44.590405 18235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.591838 18157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:44.591986 18157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "7de11838adb1ae9eaab01afd52243db7"
server_key_iv: "f504f7b718bf7b7b82bc55dba79dfbb1"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:44.592579 18157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:44.612452 18157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:44.615960 18157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:44.616187 18157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:44.617039 18157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:44.618079 18157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:44.618153 18157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.618218 18157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:44.618245 18157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.630002 18157 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:35439
I20260812 07:04:44.631251 18157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:44.632211 18436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:35439 every 8 connection(s)
I20260812 07:04:44.637275 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 18157
I20260812 07:04:44.637799 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:34231
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:36091
--webserver_port=37045
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:44.652549 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.633887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36091 (local address 127.27.110.193:35389)
0812 07:04:44.634922 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:44.634965 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:44.635852 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:44.637037 (+ 1185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:44.637046 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:44.637511 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:44.638157 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.638185 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.640170 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.640174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.641584 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.641596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.642881 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.642885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.643126 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.643863 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:44.643891 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:44.645705 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:44.648586 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.648592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.648607 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.648967 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.649431 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.649434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.649436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.649626 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.650511 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:44.650521 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:44.650835 (+ 314us) client_negotiation.cc:769] Sending connection context
0812 07:04:44.651051 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 07:04:44.651443 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"thread_start_us":149,"threads_started":1}
I20260812 07:04:44.654604 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.633688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:35389 (local address 127.27.110.254:36091)
0812 07:04:44.634322 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:44.634348 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:44.635144 (+ 796us) server_negotiation.cc:407] Connection header received
0812 07:04:44.636314 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:44.636325 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:44.636626 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:44.636965 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:44.638351 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.639980 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.641735 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.642476 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.642645 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.645946 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:44.645953 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:44.645971 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:44.646006 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:44.648429 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.649135 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.649141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.649147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.649305 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.649749 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.649753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.649755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.650281 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:44.650426 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:44.653328 (+ 2902us) server_negotiation.cc:299] Negotiation successful
0812 07:04:44.653659 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":113,"threads_started":1}
I20260812 07:04:44.657691 18437 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36091
I20260812 07:04:44.657961 18437 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.658607 18437 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
W20260812 07:04:44.661146 18437 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:36091 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 07:04:44.661278 18437 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.661466 18437 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:44.662452 18437 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.662631 18437 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
W20260812 07:04:44.663518 18437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:04:44.795580 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:44.795982 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:44.796070 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:44.802091 18443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:44.802173 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:44.802261 18443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:04:44.810545 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:34231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=37045
--tserver_master_addrs=127.27.110.254:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:44.812139 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:44.813644 18443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:44.827088 18453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:44.827378 18443 server_base.cc:1034] running on GCE node
W20260812 07:04:44.827530 18452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:44.828415 18455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:44.828922 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:44.829674 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:44.830870 18443 hybrid_clock.cc:648] HybridClock initialized: now 1786518284830854 us; error 39 us; skew 500 ppm
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:44.834143 18443 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:04:44.835139 18443 webserver.cc:460] Webserver started at http://127.27.110.194:37045/ using document root <none> and password file <none>
I20260812 07:04:44.835801 18443 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:44.835919 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:44.855734 18443 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:44.859965 18469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.861487 18443 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260812 07:04:44.861641 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "22220d415e0341c19e9bd8cd333b1962"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "e2f90102b845490105ce2e98871f9972"
server_key_iv: "771cac7c09ba70c02ef68b9981d3f8fe"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:44.862414 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:44.896116 18443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:44.900245 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:44.900496 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:44.901299 18443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:44.902406 18443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:44.902479 18443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.902550 18443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:44.902583 18443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:44.913584 18443 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:34231
I20260812 07:04:44.914939 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:44.916697 18689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:34231 every 8 connection(s)
I20260812 07:04:44.923628 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 18443
I20260812 07:04:44.924343 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.110.212:47439/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:33185
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:36091
--webserver_port=43359
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:44 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518284, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:44.936249 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.921445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:57237 (local address 127.27.110.254:36091)
0812 07:04:44.921622 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:44.921628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:44.921649 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:04:44.922083 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:44.922089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:44.922160 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:44.922271 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:44.923616 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.924286 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.926063 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.926418 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.926512 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.931236 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:44.931241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:44.931244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:44.931277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:44.933184 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.933950 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.933954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.933956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.934053 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:44.934530 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:44.934534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:44.934536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:44.934799 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:44.934913 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:44.935749 (+ 836us) server_negotiation.cc:299] Negotiation successful
0812 07:04:44.935923 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:04:44.938759 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:44.920218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36091 (local address 127.27.110.194:57237)
0812 07:04:44.920836 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:44.920885 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:44.921813 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:44.922442 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:44.922448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:44.922761 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:44.923436 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.923451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.924456 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.924461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.925884 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:44.925896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:44.926748 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:44.926752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:44.926978 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:44.927753 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:44.927779 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:44.931003 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:44.933334 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.933344 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.933368 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.933808 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.934172 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:44.934179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:44.934184 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:44.934411 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:44.934931 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:44.934942 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:44.935398 (+ 456us) client_negotiation.cc:769] Sending connection context
0812 07:04:44.936657 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0812 07:04:44.937112 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":161,"threads_started":1}
I20260812 07:04:44.940757 18691 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36091
I20260812 07:04:44.941054 18691 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.941733 18691 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
W20260812 07:04:44.943594 18691 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:36091 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 07:04:44.943715 18691 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.943871 18691 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:44.944664 18691 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:44.944823 18691 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
W20260812 07:04:44.945554 18691 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:04:45.073750 18699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:45.074061 18699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:45.074146 18699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:45.078774 18699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:45.078860 18699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:45.078946 18699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:04:45.085819 18699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.110.212:47439/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:33185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=43359
--tserver_master_addrs=127.27.110.254:36091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:45.087179 18699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:45.088423 18699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:45.099391 18711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:45.099620 18699 server_base.cc:1034] running on GCE node
W20260812 07:04:45.099254 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:45.099251 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:45.100268 18699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:45.100965 18699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:45.102188 18699 hybrid_clock.cc:648] HybridClock initialized: now 1786518285102149 us; error 48 us; skew 500 ppm
Aug 12 07:04:45 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:45.105487 18699 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:04:45.106719 18699 webserver.cc:460] Webserver started at http://127.27.110.195:43359/ using document root <none> and password file <none>
I20260812 07:04:45.107426 18699 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:45.107578 18699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 07:04:45 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:45.152915 18699 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.000s sys 0.003s
I20260812 07:04:45.156455 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:45.158254 18699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:04:45.158469 18699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "1cfa703c0cef407588f22b842b9478c8"
format_stamp: "Formatted at 2026-08-12 07:04:43 on dist-test-slave-w4v5"
server_key: "07a87080bccedb6fa28e792c28f3c6a3"
server_key_iv: "f4cd07775eab82fed4d9bee0c22f0cf8"
server_key_version: "kuduclusterkey@0"
I20260812 07:04:45.159373 18699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:45.181248 18699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:45.184527 18699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:45.184747 18699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:45.185431 18699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:45.186400 18699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:45.186475 18699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:45.186542 18699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:45.186600 18699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:45.200446 18699 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:33185
I20260812 07:04:45.201809 18699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:45.203263 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 18699
I20260812 07:04:45.211740 18927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:33185 every 8 connection(s)
I20260812 07:04:45.215173 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.204005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.27.110.254:36091)
0812 07:04:45.204145 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:45.204150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:45.206458 (+ 2308us) server_negotiation.cc:407] Connection header received
0812 07:04:45.206620 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:45.206624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:45.206680 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:45.206758 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:45.207735 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.208267 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.209483 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.209740 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.209859 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.211682 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:45.211686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:45.211689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:45.211724 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:45.213302 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.213793 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.213798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.213800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.213894 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.214207 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.214210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.214212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.214462 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:45.214703 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:45.214843 (+ 140us) server_negotiation.cc:299] Negotiation successful
0812 07:04:45.214970 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 12 07:04:45 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:45.233487 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.205981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:39815 (local address 127.27.110.254:36091)
0812 07:04:45.206734 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:45.206747 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:45.218142 (+ 11395us) server_negotiation.cc:407] Connection header received
0812 07:04:45.219439 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:45.219443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:45.219495 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:45.219590 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:45.221179 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.222094 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.223935 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.224285 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.224356 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.228893 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:45.228897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:45.228900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:45.228927 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:45.230789 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.231501 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.231505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.231507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.231599 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:45.232019 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:45.232023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:45.232025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:45.232288 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:45.232372 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:45.233141 (+ 769us) server_negotiation.cc:299] Negotiation successful
0812 07:04:45.233291 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":109,"threads_started":1}
I20260812 07:04:45.235410 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:45.204930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36091 (local address 127.27.110.195:39815)
0812 07:04:45.211047 (+ 6117us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:45.211149 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:45.219036 (+ 7887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:45.219932 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:45.219943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:45.220312 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:45.221008 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:45.221026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.222262 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.222265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:45.223800 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:45.223812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:45.224557 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:45.224572 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:45.224773 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:45.226775 (+ 2002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:45.226799 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:45.228687 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:45.230936 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:45.230942 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:45.230963 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:45.231374 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:45.231711 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:45.231714 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:45.231716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:45.231897 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:45.232446 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:45.232454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:45.232790 (+ 336us) client_negotiation.cc:769] Sending connection context
0812 07:04:45.233744 (+ 954us) client_negotiation.cc:240] Negotiation successful
0812 07:04:45.234143 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5836,"thread_start_us":162,"threads_started":1}
I20260812 07:04:45.236848 18929 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36091
I20260812 07:04:45.237128 18929 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:45.237754 18929 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
W20260812 07:04:45.239550 18929 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:36091 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 07:04:45.239697 18929 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:45.239894 18929 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:45.240970 18929 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:45.241194 18929 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
W20260812 07:04:45.242039 18929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 07:04:45 dist-test-slave-w4v5 krb5kdc[16376](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:45.663753 18437 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:45.664057 18437 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:45.665167 18094 ts_manager.cc:194] Registered new tserver with Master: dd35df0700b5429fa56a2e5d3a2d0e7e (127.27.110.193:35439)
I20260812 07:04:45.666589 18094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:35389
I20260812 07:04:45.945832 18691 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:45.946172 18691 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:45.947253 18094 ts_manager.cc:194] Registered new tserver with Master: 22220d415e0341c19e9bd8cd333b1962 (127.27.110.194:34231)
I20260812 07:04:45.948036 18094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:57237
Aug 12 07:04:46 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:46.242326 18929 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:46.242646 18929 heartbeater.cc:511] Master 127.27.110.254:36091 requested a full tablet report, sending...
I20260812 07:04:46.243487 18094 ts_manager.cc:194] Registered new tserver with Master: 1cfa703c0cef407588f22b842b9478c8 (127.27.110.195:33185)
I20260812 07:04:46.244362 18094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:39815
I20260812 07:04:46.251385 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:04:46.263875 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.253186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.27.110.254:36091)
0812 07:04:46.253948 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.253953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.254311 (+ 358us) server_negotiation.cc:407] Connection header received
0812 07:04:46.254355 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.254358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.254409 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.254485 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:46.255490 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.256357 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.257554 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.257936 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.258029 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.259921 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:46.259925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:46.259928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:46.259957 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:46.261860 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:46.262345 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:46.262348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:46.262350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:46.262438 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:46.262752 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:46.262755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:46.262757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:46.262977 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:46.263117 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.263481 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.263666 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":102,"threads_started":1}
I20260812 07:04:46.268626 18094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:46.273744 18094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 12 07:04:46 dist-test-slave-w4v5 krb5kdc[16376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for HTTP/127.27.110.212@KRBTEST.COM
I20260812 07:04:46.307524 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.293477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:34231 (local address 127.0.0.1:58872)
0812 07:04:46.294348 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:46.294362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:46.294449 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:46.301871 (+ 7422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:46.301875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:46.301897 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:46.302002 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.302009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.303677 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.303681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.304857 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.304869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.306870 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.306874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.307011 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.307065 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 07:04:46.307206 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 07:04:46.307357 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":138,"threads_started":1}
I20260812 07:04:46.309911 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.294369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.27.110.194:34231)
0812 07:04:46.300832 (+ 6463us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.300840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.300867 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:04:46.300951 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.300958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.301194 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.302191 (+ 997us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:46.302353 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.303534 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.305842 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.306713 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.309377 (+ 2664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.309542 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.309619 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.309731 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6334,"thread_start_us":100,"threads_started":1}
I20260812 07:04:46.313694 18555 tablet_service.cc:1467] Processing CreateTablet for tablet de6997984f7542fd939bf068a5ee1c4c (DEFAULT_TABLE table=test-table [id=f304117e06e0482d9992bc031a40ac00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:46.315534 18555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6997984f7542fd939bf068a5ee1c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:46.315653 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.293454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:35439 (local address 127.0.0.1:43926)
0812 07:04:46.293912 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:46.293962 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:46.294110 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:46.295726 (+ 1616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:46.295733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:46.295761 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:46.295927 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.295934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.297744 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.297748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.299310 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.299323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.315155 (+ 15832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.315162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.315273 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.315299 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:04:46.315387 (+ 88us) client_negotiation.cc:240] Negotiation successful
0812 07:04:46.315476 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":168,"threads_started":1}
I20260812 07:04:46.316372 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.294456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.27.110.193:35439)
0812 07:04:46.294772 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.294780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.294806 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:04:46.294887 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.294892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.295525 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.296111 (+ 586us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:46.296251 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.297602 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.299476 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.313434 (+ 13958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.313592 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.313733 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.316102 (+ 2369us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.316229 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":137,"threads_started":1}
I20260812 07:04:46.319355 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.295104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:33185 (local address 127.0.0.1:57356)
0812 07:04:46.307732 (+ 12628us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:46.307748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:46.307825 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:46.314620 (+ 6795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:46.314624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:46.314646 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:46.314752 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.314758 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.316470 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.316473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.317694 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.317706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.318885 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.318889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.318980 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.319010 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 07:04:46.319147 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 07:04:46.319225 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12569,"thread_start_us":108,"threads_started":1}
I20260812 07:04:46.319397 18288 tablet_service.cc:1467] Processing CreateTablet for tablet de6997984f7542fd939bf068a5ee1c4c (DEFAULT_TABLE table=test-table [id=f304117e06e0482d9992bc031a40ac00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:46.320581 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6997984f7542fd939bf068a5ee1c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:46.320652 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.295264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.27.110.195:33185)
0812 07:04:46.314077 (+ 18813us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.314084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.314111 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:04:46.314220 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.314227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.314445 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.314925 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:46.315098 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.316328 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.317860 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.318731 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.318859 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.318980 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.320399 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.320506 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18681,"thread_start_us":85,"threads_started":1}
I20260812 07:04:46.323506 18815 tablet_service.cc:1467] Processing CreateTablet for tablet de6997984f7542fd939bf068a5ee1c4c (DEFAULT_TABLE table=test-table [id=f304117e06e0482d9992bc031a40ac00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:46.324596 18815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6997984f7542fd939bf068a5ee1c4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:46.328888 18988 tablet_bootstrap.cc:492] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Bootstrap starting.
I20260812 07:04:46.332538 18988 tablet_bootstrap.cc:654] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:46.333488 18988 log.cc:824] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:46.334273 18989 tablet_bootstrap.cc:492] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: Bootstrap starting.
I20260812 07:04:46.336769 18989 tablet_bootstrap.cc:654] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:46.337718 18989 log.cc:824] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:46.338690 18988 tablet_bootstrap.cc:492] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: No bootstrap required, opened a new log
I20260812 07:04:46.338963 18988 ts_tablet_manager.cc:1397] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260812 07:04:46.340574 18989 tablet_bootstrap.cc:492] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: No bootstrap required, opened a new log
I20260812 07:04:46.340775 18989 ts_tablet_manager.cc:1397] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:04:46.344952 18988 raft_consensus.cc:348] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.344952 18989 raft_consensus.cc:348] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.345250 18988 raft_consensus.cc:374] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:46.345255 18989 raft_consensus.cc:374] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:46.345306 18988 raft_consensus.cc:729] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22220d415e0341c19e9bd8cd333b1962, State: Initialized, Role: FOLLOWER
I20260812 07:04:46.345312 18989 raft_consensus.cc:729] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd35df0700b5429fa56a2e5d3a2d0e7e, State: Initialized, Role: FOLLOWER
I20260812 07:04:46.345793 18989 consensus_queue.cc:260] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.346048 18992 tablet_bootstrap.cc:492] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Bootstrap starting.
I20260812 07:04:46.346992 18989 ts_tablet_manager.cc:1428] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:04:46.347697 18437 heartbeater.cc:503] Master 127.27.110.254:36091 was elected leader, sending a full tablet report...
I20260812 07:04:46.348501 18992 tablet_bootstrap.cc:654] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:46.349406 18992 log.cc:824] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:46.350188 18988 consensus_queue.cc:260] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.351411 18988 ts_tablet_manager.cc:1428] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260812 07:04:46.351828 18691 heartbeater.cc:503] Master 127.27.110.254:36091 was elected leader, sending a full tablet report...
I20260812 07:04:46.357475 18992 tablet_bootstrap.cc:492] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: No bootstrap required, opened a new log
I20260812 07:04:46.357726 18992 ts_tablet_manager.cc:1397] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260812 07:04:46.362037 18992 raft_consensus.cc:348] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.362417 18992 raft_consensus.cc:374] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:46.362502 18992 raft_consensus.cc:729] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa703c0cef407588f22b842b9478c8, State: Initialized, Role: FOLLOWER
I20260812 07:04:46.362999 18992 consensus_queue.cc:260] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.364252 18992 ts_tablet_manager.cc:1428] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:04:46.365098 18929 heartbeater.cc:503] Master 127.27.110.254:36091 was elected leader, sending a full tablet report...
W20260812 07:04:46.383674 18440 tablet.cc:2369] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:46.419127 18693 tablet.cc:2369] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:46.442278 18999 raft_consensus.cc:482] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:46.442541 18999 raft_consensus.cc:504] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.444031 18999 leader_election.cc:288] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22220d415e0341c19e9bd8cd333b1962 (127.27.110.194:34231), 1cfa703c0cef407588f22b842b9478c8 (127.27.110.195:33185)
W20260812 07:04:46.453951 18930 tablet.cc:2369] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:46.459159 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.445322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:49881 (local address 127.27.110.195:33185)
0812 07:04:46.445441 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.445445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.454406 (+ 8961us) server_negotiation.cc:407] Connection header received
0812 07:04:46.454605 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.454608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.454667 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.454718 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:46.455183 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.455942 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.457635 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.458295 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.458432 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.458495 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.458887 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.458977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:04:46.459654 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.444915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:33185 (local address 127.27.110.193:49881)
0812 07:04:46.454306 (+ 9391us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:46.454323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:46.454479 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:46.454857 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:46.454860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:46.454877 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:46.455000 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.455019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.456112 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.456115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.457472 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.457484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.458605 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.458609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.458725 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.458751 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 07:04:46.459404 (+ 653us) client_negotiation.cc:240] Negotiation successful
0812 07:04:46.459489 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9316,"thread_start_us":120,"threads_started":1}
I20260812 07:04:46.460466 18831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6997984f7542fd939bf068a5ee1c4c" candidate_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa703c0cef407588f22b842b9478c8" is_pre_election: true
I20260812 07:04:46.460851 18831 raft_consensus.cc:2454] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd35df0700b5429fa56a2e5d3a2d0e7e in term 0.
I20260812 07:04:46.461491 18242 leader_election.cc:302] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfa703c0cef407588f22b842b9478c8, dd35df0700b5429fa56a2e5d3a2d0e7e; no voters:
I20260812 07:04:46.461848 18999 raft_consensus.cc:2781] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:46.461927 18999 raft_consensus.cc:482] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:46.461966 18999 raft_consensus.cc:3037] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:46.463461 18999 raft_consensus.cc:504] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.463879 18999 leader_election.cc:288] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [CANDIDATE]: Term 1 election: Requested vote from peers 22220d415e0341c19e9bd8cd333b1962 (127.27.110.194:34231), 1cfa703c0cef407588f22b842b9478c8 (127.27.110.195:33185)
I20260812 07:04:46.465713 18831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6997984f7542fd939bf068a5ee1c4c" candidate_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa703c0cef407588f22b842b9478c8"
I20260812 07:04:46.465873 18831 raft_consensus.cc:3037] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:46.467299 18831 raft_consensus.cc:2454] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd35df0700b5429fa56a2e5d3a2d0e7e in term 1.
I20260812 07:04:46.467751 18242 leader_election.cc:302] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfa703c0cef407588f22b842b9478c8, dd35df0700b5429fa56a2e5d3a2d0e7e; no voters:
I20260812 07:04:46.467975 18999 raft_consensus.cc:2781] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:46.468223 18999 raft_consensus.cc:686] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [term 1 LEADER]: Becoming Leader. State: Replica: dd35df0700b5429fa56a2e5d3a2d0e7e, State: Running, Role: LEADER
I20260812 07:04:46.468622 18999 consensus_queue.cc:237] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } }
I20260812 07:04:46.472191 18093 catalog_manager.cc:5495] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e reported cstate change: term changed from 0 to 1, leader changed from <none> to dd35df0700b5429fa56a2e5d3a2d0e7e (127.27.110.193). New cstate: current_term: 1 leader_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 35439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:46.474224 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.444553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:40379 (local address 127.27.110.194:34231)
0812 07:04:46.444694 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.444698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.451321 (+ 6623us) server_negotiation.cc:407] Connection header received
0812 07:04:46.451610 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.451614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.451671 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.451721 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:46.452316 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.453156 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.472696 (+ 19540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.473343 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.473455 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.473522 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.473997 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.474095 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:04:46.474897 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.444465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:34231 (local address 127.27.110.193:40379)
0812 07:04:46.451184 (+ 6719us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:46.451205 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:46.451429 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:46.451913 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:46.451916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:46.451935 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:46.452170 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.452178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.453315 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.453318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.472491 (+ 19173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:46.472517 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.473720 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.473724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:46.473833 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.473861 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:04:46.474643 (+ 782us) client_negotiation.cc:240] Negotiation successful
0812 07:04:46.474741 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6582,"thread_start_us":131,"threads_started":1}
I20260812 07:04:46.475749 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6997984f7542fd939bf068a5ee1c4c" candidate_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220d415e0341c19e9bd8cd333b1962" is_pre_election: true
I20260812 07:04:46.476120 18592 raft_consensus.cc:2454] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd35df0700b5429fa56a2e5d3a2d0e7e in term 0.
I20260812 07:04:46.476490 18590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6997984f7542fd939bf068a5ee1c4c" candidate_uuid: "dd35df0700b5429fa56a2e5d3a2d0e7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22220d415e0341c19e9bd8cd333b1962"
I20260812 07:04:46.476611 18590 raft_consensus.cc:3037] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:46.478186 18590 raft_consensus.cc:2454] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd35df0700b5429fa56a2e5d3a2d0e7e in term 1.
I20260812 07:04:46.525496 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.520047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.27.110.193:35439)
0812 07:04:46.520503 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:46.520507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:46.520523 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:04:46.520636 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:46.520640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:46.520707 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:46.520814 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:46.521197 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.521996 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.523235 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:46.523545 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:46.523648 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:46.524064 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:46.524673 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:46.524798 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:46.525262 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 07:04:46.525327 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366}
I20260812 07:04:46.536505 18831 raft_consensus.cc:1261] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 1 FOLLOWER]: Refusing update from remote peer dd35df0700b5429fa56a2e5d3a2d0e7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:46.536501 18590 raft_consensus.cc:1261] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 1 FOLLOWER]: Refusing update from remote peer dd35df0700b5429fa56a2e5d3a2d0e7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:46.537369 18999 consensus_queue.cc:1035] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "22220d415e0341c19e9bd8cd333b1962" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:46.537621 19010 consensus_queue.cc:1035] T de6997984f7542fd939bf068a5ee1c4c P dd35df0700b5429fa56a2e5d3a2d0e7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa703c0cef407588f22b842b9478c8" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:46.553648 19017 mvcc.cc:205] Tried to move back new op lower bound from 7317578901643894784 to 7317578901379219456. Current Snapshot: MvccSnapshot[applied={T|T < 7317578901643894784}]
I20260812 07:04:46.555390 19025 mvcc.cc:205] Tried to move back new op lower bound from 7317578901643894784 to 7317578901379219456. Current Snapshot: MvccSnapshot[applied={T|T < 7317578901643894784}]
I20260812 07:04:46.557873 19022 mvcc.cc:205] Tried to move back new op lower bound from 7317578901643894784 to 7317578901379219456. Current Snapshot: MvccSnapshot[applied={T|T < 7317578901643894784}]
I20260812 07:04:46.569707 18093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:46.569927 18093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:46.573251 18093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ace375dbb4d4f569a3aef22fdd11598: Sending DeleteTablet for 3 replicas of tablet de6997984f7542fd939bf068a5ee1c4c
I20260812 07:04:46.574455 18815 tablet_service.cc:1514] Processing DeleteTablet for tablet de6997984f7542fd939bf068a5ee1c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:46 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:57356
I20260812 07:04:46.575456 18555 tablet_service.cc:1514] Processing DeleteTablet for tablet de6997984f7542fd939bf068a5ee1c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:46 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:58872
I20260812 07:04:46.576648 18288 tablet_service.cc:1514] Processing DeleteTablet for tablet de6997984f7542fd939bf068a5ee1c4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:46 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43926
I20260812 07:04:46.577926 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 18157
I20260812 07:04:46.593837 19038 tablet_replica.cc:330] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: stopping tablet replica
I20260812 07:04:46.594235 19039 tablet_replica.cc:330] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: stopping tablet replica
I20260812 07:04:46.603556 19039 raft_consensus.cc:2229] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:46.604120 19038 raft_consensus.cc:2229] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:46.604641 19038 raft_consensus.cc:2258] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:46.605036 19039 raft_consensus.cc:2258] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:46.607847 19038 ts_tablet_manager.cc:1905] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:46.609628 19039 ts_tablet_manager.cc:1905] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:46.611050 19038 ts_tablet_manager.cc:1918] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:46.611189 19038 log.cc:1196] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/de6997984f7542fd939bf068a5ee1c4c
I20260812 07:04:46.611704 19038 ts_tablet_manager.cc:1939] T de6997984f7542fd939bf068a5ee1c4c P 1cfa703c0cef407588f22b842b9478c8: Deleting consensus metadata
I20260812 07:04:46.613184 19039 ts_tablet_manager.cc:1918] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:46.613296 19039 log.cc:1196] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/de6997984f7542fd939bf068a5ee1c4c
I20260812 07:04:46.613238 18075 catalog_manager.cc:4841] TS 1cfa703c0cef407588f22b842b9478c8 (127.27.110.195:33185): tablet de6997984f7542fd939bf068a5ee1c4c (table test-table [id=f304117e06e0482d9992bc031a40ac00]) successfully deleted
I20260812 07:04:46.613637 19039 ts_tablet_manager.cc:1939] T de6997984f7542fd939bf068a5ee1c4c P 22220d415e0341c19e9bd8cd333b1962: Deleting consensus metadata
I20260812 07:04:46.614984 18072 catalog_manager.cc:4841] TS 22220d415e0341c19e9bd8cd333b1962 (127.27.110.194:34231): tablet de6997984f7542fd939bf068a5ee1c4c (table test-table [id=f304117e06e0482d9992bc031a40ac00]) successfully deleted
W20260812 07:04:46.615949 18075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:35439 (error 108)
I20260812 07:04:46.616935 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 18443
I20260812 07:04:46.617105 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:46.616751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:35439 (local address 127.0.0.1:43936)
0812 07:04:46.616935 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:46.617010 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:35439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260812 07:04:46.626061 18075 catalog_manager.cc:4584] TS dd35df0700b5429fa56a2e5d3a2d0e7e (127.27.110.193:35439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de6997984f7542fd939bf068a5ee1c4c: Network error: Client connection negotiation failed: client connection to 127.27.110.193:35439: connect: Connection refused (error 111)
I20260812 07:04:46.633029 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 18699
I20260812 07:04:46.658746 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 18043
I20260812 07:04:46.680457 28091 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 07:04:47.027793 28091 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 07:04:47.027915 28091 mini_ranger.cc:66] Stopping Ranger...
I20260812 07:04:47.383307 28091 mini_ranger.cc:68] Stopped Ranger
2026-08-12 07:04:47.383 UTC [16432] LOG: received smart shutdown request
2026-08-12 07:04:47.387 UTC [16432] LOG: background worker "logical replication launcher" (PID 16454) exited with exit code 1
2026-08-12 07:04:47.388 UTC [16442] LOG: shutting down
2026-08-12 07:04:47.534 UTC [16432] LOG: database system is shut down
2026-08-12T07:04:47Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51044 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:47 dist-test-slave-w4v5 krb5kdc[19078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:47 dist-test-slave-w4v5 krb5kdc[19078](info): set up 2 sockets
Aug 12 07:04:47 dist-test-slave-w4v5 krb5kdc[19078](info): commencing operation
krb5kdc: starting...
W20260812 07:04:48.962970 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:04:49.090257 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:37665
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:37665,127.27.110.253:44679,127.27.110.252:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:49.205072 19095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:49.205375 19095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:49.205447 19095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:49.209417 19095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:49.209522 19095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:49.209564 19095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:49.209625 19095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:49.209681 19095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:49.215139 19095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:37665,127.27.110.253:44679,127.27.110.252:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:49.216264 19095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:49.217391 19095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:49.226143 19104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:49.226333 19095 server_base.cc:1034] running on GCE node
W20260812 07:04:49.226125 19105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:49.226167 19108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:49.227128 19095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:49.228207 19095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:49.229421 19095 hybrid_clock.cc:648] HybridClock initialized: now 1786518289229363 us; error 70 us; skew 500 ppm
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:49.232534 19095 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:49.233451 19095 webserver.cc:460] Webserver started at http://127.27.110.254:39679/ using document root <none> and password file <none>
I20260812 07:04:49.234031 19095 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:49.234127 19095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:49.234372 19095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:49.236529 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "64b3af9aa0f545aaa9ff16652a83a4fe"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "bb731393926144dcd014d458485c8379"
server_key_iv: "d4560748237fe50eb88d90c7a19adaf3"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.237144 19095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "64b3af9aa0f545aaa9ff16652a83a4fe"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "bb731393926144dcd014d458485c8379"
server_key_iv: "d4560748237fe50eb88d90c7a19adaf3"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.241021 19095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:04:49.243628 19120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:49.244800 19095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:04:49.244944 19095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "64b3af9aa0f545aaa9ff16652a83a4fe"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "bb731393926144dcd014d458485c8379"
server_key_iv: "d4560748237fe50eb88d90c7a19adaf3"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.245060 19095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:49.265360 19095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:49.268508 19095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:49.268728 19095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:49.275763 19095 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:37665
I20260812 07:04:49.276072 19212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:37665 every 8 connection(s)
I20260812 07:04:49.276768 19095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:49.279719 19213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:49.283493 19213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.287007 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 19095
I20260812 07:04:49.287184 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:04:49.287494 28091 external_mini_cluster.cc:1428] Setting key 915939b9b84b6ef6fa3efe726276a953
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.253@KRBTEST.COM; defaulting to no policy
I20260812 07:04:49.298885 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.285725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40504 (local address 127.27.110.254:37665)
0812 07:04:49.286249 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.286264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.286412 (+ 148us) server_negotiation.cc:407] Connection header received
0812 07:04:49.287634 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.287639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.287878 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.288087 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.288895 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.290138 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.291192 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.291562 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.291740 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.294149 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.294155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.294165 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.294185 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.295899 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.296372 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.296379 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.296386 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.296487 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.296809 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.296814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.296816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.297246 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.297390 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.297755 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.298011 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":406,"spinlock_wait_cycles":22272,"thread_start_us":179,"threads_started":1}
I20260812 07:04:49.298888 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.285725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37665 (local address 127.0.0.1:40504)
0812 07:04:49.286247 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.286300 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.287308 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.288102 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.288107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.288457 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.288728 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.288742 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.290282 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.290285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.291043 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.291053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.291726 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.291730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.291844 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.292486 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.292503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.293992 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.296010 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.296020 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.296030 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.296261 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.296583 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.296588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.296590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.296715 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.297360 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.297367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.297515 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.297694 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.298146 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":136,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.253@KRBTEST.COM" created.
I20260812 07:04:49.301226 19213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.301976 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.301706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46426)
0812 07:04:49.301846 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.301919 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 07:04:49.302186 19123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
I20260812 07:04:49.303030 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.302815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46438)
0812 07:04:49.302926 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.302977 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 07:04:49.303227 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:44679: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab.
Entry for principal kudu/127.27.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab.
I20260812 07:04:49.316239 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:44679
--webserver_interface=127.27.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab
--principal=kudu/127.27.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:37665,127.27.110.253:44679,127.27.110.252:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:49.352499 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } attempt: 1
I20260812 07:04:49.353389 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.353070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46446)
0812 07:04:49.353235 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.353307 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 07:04:49.354228 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.354011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46458)
0812 07:04:49.354115 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.354163 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 07:04:49.354446 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:44679: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
I20260812 07:04:49.422637 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } attempt: 2
I20260812 07:04:49.423516 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.423210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46470)
0812 07:04:49.423367 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.423439 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:04:49.424314 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.424072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46486)
0812 07:04:49.424201 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.424256 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 07:04:49.424526 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:44679: Network error: Client connection negotiation failed: client connection to 127.27.110.253:44679: connect: Connection refused (error 111)
W20260812 07:04:49.430420 19225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:49.430696 19225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:49.430775 19225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:49.434890 19225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:49.434942 19225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:49.434962 19225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:49.435017 19225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:49.435052 19225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:49.440352 19225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:37665,127.27.110.253:44679,127.27.110.252:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab
--principal=kudu/127.27.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:44679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:49.441432 19225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:49.442524 19225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:49.451171 19234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:49.451190 19233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:49.451340 19225 server_base.cc:1034] running on GCE node
W20260812 07:04:49.451190 19239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:49.452109 19225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:49.453158 19225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:49.454329 19225 hybrid_clock.cc:648] HybridClock initialized: now 1786518289454295 us; error 46 us; skew 500 ppm
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:49.457131 19225 init.cc:377] Logged in from keytab as kudu/127.27.110.253@KRBTEST.COM (short username kudu)
I20260812 07:04:49.458001 19225 webserver.cc:460] Webserver started at http://127.27.110.253:38917/ using document root <none> and password file <none>
I20260812 07:04:49.458560 19225 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:49.458621 19225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:49.458876 19225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:49.460971 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data/instance:
uuid: "ae51839e4c0a4cfabae9db793c894548"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "0e4adbb0c8434b7f1e7df17d5d626a6e"
server_key_iv: "83a11f5e2d65074652367076eb77e189"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.461546 19225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal/instance:
uuid: "ae51839e4c0a4cfabae9db793c894548"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "0e4adbb0c8434b7f1e7df17d5d626a6e"
server_key_iv: "83a11f5e2d65074652367076eb77e189"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.465366 19225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 07:04:49.467796 19249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:49.468997 19225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:49.469141 19225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal
uuid: "ae51839e4c0a4cfabae9db793c894548"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "0e4adbb0c8434b7f1e7df17d5d626a6e"
server_key_iv: "83a11f5e2d65074652367076eb77e189"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.469264 19225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:49.486363 19225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:49.489439 19225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:49.489655 19225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:49.496755 19225 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.253:44679
I20260812 07:04:49.497061 19345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.253:44679 every 8 connection(s)
I20260812 07:04:49.497732 19225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
I20260812 07:04:49.500702 19346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:49.502962 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 19225
I20260812 07:04:49.503136 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-1/wal/instance
I20260812 07:04:49.503418 28091 external_mini_cluster.cc:1428] Setting key 2460f19ae26961553457db5777484044
I20260812 07:04:49.504643 19346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.252@KRBTEST.COM" created.
I20260812 07:04:49.515717 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } attempt: 3
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:04:49.522795 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.506884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37665 (local address 127.0.0.1:40520)
0812 07:04:49.507416 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.507470 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.508357 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.508951 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.508958 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.509309 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.509799 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.509812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.511084 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.511090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.511955 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.511965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.512538 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.512543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.512768 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.513398 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.513415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.516810 (+ 3395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.519696 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.519703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.519718 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.520137 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.520434 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.520438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.520440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.520587 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.521064 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.521073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.521429 (+ 356us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.521669 (+ 240us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.521945 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":193,"threads_started":1}
I20260812 07:04:49.522864 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.506543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.27.110.254:37665)
0812 07:04:49.506738 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.506742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.507592 (+ 850us) server_negotiation.cc:407] Connection header received
0812 07:04:49.508614 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.508620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.508687 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.508799 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.509957 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.510863 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.512107 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.512369 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.512489 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.517013 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.517017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.517019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.517051 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.519568 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.520259 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.520263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.520264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.520340 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.520690 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.520693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.520695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.520936 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.521333 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.522555 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.522701 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:04:49.524410 19346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.527163 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.516207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46498)
0812 07:04:49.516366 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.516382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.516469 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.517533 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.517537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.517764 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.517935 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.517946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.519182 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.519186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.519776 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.519790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.520789 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.520793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.520900 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.521339 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.521356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.522924 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.525343 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.525345 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.525347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.525595 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.525930 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.525934 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.525937 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.526031 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.526623 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.526627 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.526725 (+ 98us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.526832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.526975 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:04:49.527478 19354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.516636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.27.110.253:44679)
0812 07:04:49.517060 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.517075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.517103 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:04:49.517152 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.517160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.517367 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.517516 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.518071 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.518985 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.519916 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.520606 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.520791 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.523131 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.523138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.523151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.523175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.525220 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.525707 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.525715 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.525720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.525820 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.526156 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.526162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.526167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.526482 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.526648 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.526962 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.527245 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":285,"thread_start_us":166,"threads_started":1}
I20260812 07:04:49.529444 19213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.530367 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.530004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41256)
0812 07:04:49.530205 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.530280 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:04:49.531312 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.531035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41272)
0812 07:04:49.531206 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.531260 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260812 07:04:49.531531 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.534859 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.524931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46508)
0812 07:04:49.525404 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.525416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.525496 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.525823 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.525824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.526005 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.526127 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.526137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.526954 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.526957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.527799 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.527809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.528909 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.528913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.529018 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.529515 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.529530 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.531149 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.533151 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.533154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.533155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.533344 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.533728 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.533730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.533732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.533817 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.534332 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.534340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.534437 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.534614 (+ 177us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.534747 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":22}
I20260812 07:04:49.535032 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.524960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.27.110.253:44679)
0812 07:04:49.525232 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.525239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.525479 (+ 240us) server_negotiation.cc:407] Connection header received
0812 07:04:49.525593 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.525596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.525649 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.525732 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.526216 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.526847 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.527915 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.528766 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.528876 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.531306 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.531309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.531312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.531340 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.533046 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.533491 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.533495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.533497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.533598 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.533920 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.533926 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.533929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.534197 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.534298 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.534682 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.534808 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260812 07:04:49.535701 19346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.536473 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.536212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41276)
0812 07:04:49.536338 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.536421 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 07:04:49.536722 19255 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab.
Entry for principal kudu/127.27.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab.
I20260812 07:04:49.537580 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.537371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41282)
0812 07:04:49.537477 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.537533 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 07:04:49.537806 19346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.538168 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:46547
--webserver_interface=127.27.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab
--principal=kudu/127.27.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:41629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:37665,127.27.110.253:44679,127.27.110.252:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:49.562706 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 1
I20260812 07:04:49.563576 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.563289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41296)
0812 07:04:49.563430 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.563500 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 07:04:49.564320 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.564107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41306)
0812 07:04:49.564211 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.564265 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 07:04:49.564441 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.587105 19346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 1
I20260812 07:04:49.588021 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.587673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41322)
0812 07:04:49.587854 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.587927 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260812 07:04:49.588788 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.588535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41326)
0812 07:04:49.588676 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.588727 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260812 07:04:49.588946 19346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.639583 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 2
I20260812 07:04:49.640620 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.640314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41342)
0812 07:04:49.640466 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.640541 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 07:04:49.641605 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.641321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41348)
0812 07:04:49.641450 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.641528 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 07:04:49.641767 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.657110 19346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 2
I20260812 07:04:49.658030 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.657669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41364)
0812 07:04:49.657835 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.657911 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 07:04:49.658842 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.658619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41378)
0812 07:04:49.658725 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.658777 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 07:04:49.659008 19346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
W20260812 07:04:49.663240 19360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:49.663619 19360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:49.663703 19360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:49.670993 19360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:49.671062 19360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:49.671137 19360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:49.671173 19360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:49.671198 19360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:49.681115 19360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:41629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:37665,127.27.110.253:44679,127.27.110.252:46547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab
--principal=kudu/127.27.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:46547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:49.682505 19360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:49.683981 19360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:49.693827 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:49.693854 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:49.694054 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:49.695175 19360 server_base.cc:1034] running on GCE node
I20260812 07:04:49.695955 19360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:49.697592 19360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:49.698808 19360 hybrid_clock.cc:648] HybridClock initialized: now 1786518289698755 us; error 62 us; skew 500 ppm
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:49.702832 19360 init.cc:377] Logged in from keytab as kudu/127.27.110.252@KRBTEST.COM (short username kudu)
I20260812 07:04:49.704159 19360 webserver.cc:460] Webserver started at http://127.27.110.252:42413/ using document root <none> and password file <none>
I20260812 07:04:49.704931 19360 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:49.705015 19360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:49.705261 19360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:49.707614 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data/instance:
uuid: "b2b3a2bd0b2340819df6fccc592ea84f"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "d8886a3a435c27f7cc019de3a1805dd2"
server_key_iv: "6c7326e5b77d6dac2d1db3b7c3737ad3"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.708273 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal/instance:
uuid: "b2b3a2bd0b2340819df6fccc592ea84f"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "d8886a3a435c27f7cc019de3a1805dd2"
server_key_iv: "6c7326e5b77d6dac2d1db3b7c3737ad3"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.712415 19360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:04:49.714895 19393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:49.716286 19360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:04:49.716430 19360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal
uuid: "b2b3a2bd0b2340819df6fccc592ea84f"
format_stamp: "Formatted at 2026-08-12 07:04:49 on dist-test-slave-w4v5"
server_key: "d8886a3a435c27f7cc019de3a1805dd2"
server_key_iv: "6c7326e5b77d6dac2d1db3b7c3737ad3"
server_key_version: "encryptionkey@0"
I20260812 07:04:49.716547 19360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:49.740314 19360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:49.740926 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 3
I20260812 07:04:49.741789 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.741490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41380)
0812 07:04:49.741646 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.741711 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 07:04:49.742501 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.742289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41384)
0812 07:04:49.742392 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.742447 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 07:04:49.742627 19213 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.744159 19360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:49.744379 19360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:49.750248 19346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 3
I20260812 07:04:49.751169 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.750776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41394)
0812 07:04:49.750958 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.751029 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 07:04:49.752203 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.751820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41404)
0812 07:04:49.752074 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.752135 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260812 07:04:49.752251 19360 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.252:46547
W20260812 07:04:49.752358 19346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:46547: Network error: Client connection negotiation failed: client connection to 127.27.110.252:46547: connect: Connection refused (error 111)
I20260812 07:04:49.752629 19474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.252:46547 every 8 connection(s)
I20260812 07:04:49.753363 19360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
I20260812 07:04:49.756705 19475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:49.760797 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 19360
I20260812 07:04:49.760934 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786518006756923-28091-0/minicluster-data/master-2/wal/instance
I20260812 07:04:49.760799 19475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.761241 28091 external_mini_cluster.cc:1428] Setting key f2a2401069760ddde62bb7c98baa77f8
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:04:49.774803 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.763481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.27.110.254:37665)
0812 07:04:49.763615 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.763619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.763886 (+ 267us) server_negotiation.cc:407] Connection header received
0812 07:04:49.764314 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.764316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.764362 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.764427 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.766244 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.766727 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.767843 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.768060 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.768171 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.770645 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.770667 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.770672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.770704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.772397 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.773025 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.773027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.773028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.773143 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.773523 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.773527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.773529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.773776 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.773895 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.774489 (+ 594us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.774626 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:04:49.775823 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.762603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.27.110.252:46547)
0812 07:04:49.763002 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.763017 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.763050 (+ 33us) server_negotiation.cc:407] Connection header received
0812 07:04:49.764029 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.764050 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.764310 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.764502 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.765390 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.766669 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.767578 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.767967 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.768141 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.771143 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.771151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.771166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.771193 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.773139 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.773679 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.773684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.773685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.773755 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.774048 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.774051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.774052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.774307 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.774496 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.774730 (+ 234us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.774847 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260812 07:04:49.775823 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.763348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:37665 (local address 127.0.0.1:40524)
0812 07:04:49.763729 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.763785 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.764103 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.765430 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.765436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.765881 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.766089 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.766102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.766869 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.766872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.767711 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.767722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.768210 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.768214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.768320 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.768972 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.768988 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.770494 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.772526 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.772545 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.772555 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.772929 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.773267 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.773270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.773272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.773425 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.773907 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.773913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.774219 (+ 306us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.774410 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.774742 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260812 07:04:49.777426 19475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:49.786728 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.777916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:44679 (local address 127.0.0.1:46522)
0812 07:04:49.778028 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.778044 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.778120 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.778374 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.778376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.778507 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.778627 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.778633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.779768 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.779770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.780544 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.780554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.781101 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.781104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.781183 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.781696 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.781711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.783388 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.785238 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.785241 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.785243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.785438 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.785728 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.785732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.785733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.785817 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.786222 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.786226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.786319 (+ 93us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.786428 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.786564 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 07:04:49.786754 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.778055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46522 (local address 127.27.110.253:44679)
0812 07:04:49.778177 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.778182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.778201 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:04:49.778241 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.778243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.778283 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.778347 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.778785 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.779653 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.780656 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.780964 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.781092 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.783536 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.783539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.783541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.783567 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.785109 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.785538 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.785543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.785545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.785633 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.785914 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.785917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.785919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.786107 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.786220 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.786483 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.786604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:04:49.787974 19475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:04:49.788177 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.778895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.27.110.254:37665)
0812 07:04:49.779006 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.779009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.779020 (+ 11us) server_negotiation.cc:407] Connection header received
0812 07:04:49.779159 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.779162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.779219 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.779280 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.779983 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.780719 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.781706 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.782041 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.782173 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.784893 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.784896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.784898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.784919 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.786382 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.786806 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.786810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.786812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.786894 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.787224 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.787227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.787229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.787473 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.787672 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.787891 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.787996 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:04:49.796748 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.788699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41424)
0812 07:04:49.788790 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.788802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.788871 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.789082 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.789084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.789216 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.789285 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.789289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.790038 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.790043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.790873 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.790883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.791391 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.791395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.791469 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.792015 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.792026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.793119 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.795167 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.795173 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.795177 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.795423 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.795727 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.795732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.795735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.795836 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.796312 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.796318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.796428 (+ 110us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.796531 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.796636 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260812 07:04:49.796911 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.788770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.27.110.252:46547)
0812 07:04:49.788889 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.788894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.788907 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:04:49.788942 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.788944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.788983 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.789063 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.789418 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.789917 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.790980 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.791277 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.791392 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.793262 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.793265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.793266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.793286 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.794819 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.795525 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.795530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.795532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.795619 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.795937 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.795940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.795941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.796194 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.796299 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.796620 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.796740 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 07:04:49.798950 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.789417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.27.110.253:44679)
0812 07:04:49.789527 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.789531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.789554 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:04:49.789594 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.789598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.789633 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.789727 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.790462 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.791210 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.792305 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.792535 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.792645 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.795390 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.795393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.795395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.795416 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.797315 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.797783 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.797788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.797790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.797870 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.798148 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.798152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.798154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.798323 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.798564 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.798684 (+ 120us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.798798 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 07:04:49.800823 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: Bootstrap starting.
I20260812 07:04:49.802408 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:04:49.803478 19475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:49.804255 19475 log.cc:824] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:49.807406 19475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: No bootstrap required, opened a new log
I20260812 07:04:49.812080 19475 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:49.812386 19475 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:49.812451 19475 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b3a2bd0b2340819df6fccc592ea84f, State: Initialized, Role: FOLLOWER
I20260812 07:04:49.812984 19475 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:49.814231 19357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.804819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.27.110.253:44679)
0812 07:04:49.804945 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.804950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.805252 (+ 302us) server_negotiation.cc:407] Connection header received
0812 07:04:49.805366 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.805369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.805417 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.805503 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.806310 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.807001 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.808141 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.808456 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.808538 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.810542 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.810546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.810549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.810577 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.812256 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.812815 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.812819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.812820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.812909 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.813243 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.813248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.813253 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.813472 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.813587 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.813944 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.814063 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 07:04:49.814512 19475 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:49.814885 19497 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:49.815013 19497 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:49.816180 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.804676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40542 (local address 127.27.110.254:37665)
0812 07:04:49.804857 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.804863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.805207 (+ 344us) server_negotiation.cc:407] Connection header received
0812 07:04:49.805361 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.805365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.805419 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.805493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.806451 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.807268 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.808403 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.808757 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.808836 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.810635 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.810640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.810644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.810677 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.812504 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.813157 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.813163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.813166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.813260 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.813751 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.813756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.813761 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.813998 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.814104 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.815902 (+ 1798us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.816022 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:04:49.816627 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.804955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.27.110.252:46547)
0812 07:04:49.805072 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.805077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.806176 (+ 1099us) server_negotiation.cc:407] Connection header received
0812 07:04:49.806218 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.806221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.806267 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.806340 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.807205 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.807924 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.809517 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.809879 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.810001 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.812625 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.812631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.812636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.812660 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.814391 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.814900 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.814904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.814905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.814992 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.815330 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.815335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.815338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.815628 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.816242 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.816330 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.816456 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 07:04:49.820344 19519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:04:49.820427 19519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:04:49.818811 19360 master_runner.cc:186] Error getting master registration for 127.27.110.254:37665: OK, instance_id { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" instance_seqno: 1786518289273881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:04:49.822260 19360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:04:49.895511 19346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 4
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:04:49.904467 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.896055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.27.110.252:46547)
0812 07:04:49.896173 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.896176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.896188 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:04:49.896307 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.896309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.896347 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.896413 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.896918 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.897406 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.898491 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.898716 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.898789 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.901187 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.901190 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.901192 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.901214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.902735 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.903249 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.903253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.903255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.903338 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.903645 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.903648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.903655 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.903826 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.903902 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.904212 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.904311 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:04:49.904487 19352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.895947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41434)
0812 07:04:49.896108 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.896125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.896212 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.896455 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.896458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.896690 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.896805 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.896812 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.897510 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.897514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.898366 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.898378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.898821 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.898825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.898919 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.899569 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.899585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.901041 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.902848 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.902852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.902854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.903135 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.903441 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.903446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.903449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.903548 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.903923 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.903926 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.904038 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.904165 (+ 127us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.904316 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 07:04:49.906817 19213 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } attempt: 4
I20260812 07:04:49.907584 19346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: Bootstrap starting.
I20260812 07:04:49.911486 19346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: Neither blocks nor log segments found. Creating new log.
Aug 12 07:04:49 dist-test-slave-w4v5 krb5kdc[19078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518289, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:04:49.912432 19346 log.cc:824] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:49.914768 19346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: No bootstrap required, opened a new log
I20260812 07:04:49.916450 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.907274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:46547 (local address 127.0.0.1:41448)
0812 07:04:49.907454 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:49.907469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:49.907558 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:49.907878 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:49.907882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:49.908066 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:49.908177 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.908185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.909134 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.909137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.909962 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:49.909975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.910600 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.910604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:49.910712 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.911242 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:49.911257 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:49.912848 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:49.914756 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.914759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.914761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.915010 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.915345 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:49.915348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:49.915350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:49.915448 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:49.915931 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:49.915935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:49.916038 (+ 103us) client_negotiation.cc:769] Sending connection context
0812 07:04:49.916146 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 07:04:49.916279 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260812 07:04:49.916478 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:49.907327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.27.110.252:46547)
0812 07:04:49.907427 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:49.907431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:49.907515 (+ 84us) server_negotiation.cc:407] Connection header received
0812 07:04:49.907645 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:49.907647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:49.907704 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:49.907833 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:49.908306 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.909009 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.910105 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:49.910433 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:49.910949 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:49.913004 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:49.913008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:49.913010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:49.913034 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:49.914605 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.915132 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.915137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.915139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.915234 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:49.915549 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:49.915552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:49.915554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:49.915776 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:49.915920 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:49.916222 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 07:04:49.916339 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 07:04:49.918699 19346 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:49.918905 19346 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:49.918960 19346 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae51839e4c0a4cfabae9db793c894548, State: Initialized, Role: FOLLOWER
I20260812 07:04:49.919452 19346 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:49.919927 19213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: Bootstrap starting.
I20260812 07:04:49.920050 19524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:49.920230 19524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:49.920871 19346 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:49.922811 19213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:49.923969 19213 log.cc:824] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:49.926273 19213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: No bootstrap required, opened a new log
I20260812 07:04:49.924535 19225 master_runner.cc:186] Error getting master registration for 127.27.110.254:37665: OK, instance_id { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" instance_seqno: 1786518289273881 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:04:49.927747 19225 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 07:04:49.928680 19542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:04:49.928771 19542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:04:49.930485 19213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:49.930728 19213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:49.930791 19213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b3af9aa0f545aaa9ff16652a83a4fe, State: Initialized, Role: FOLLOWER
I20260812 07:04:49.931303 19213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:49.931854 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:49.932060 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:49.932715 19213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:04:49.938076 19570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:04:49.938247 19570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:04:49.944034 19095 master_runner.cc:263] No leader master found from master 64b3af9aa0f545aaa9ff16652a83a4fe
I20260812 07:04:49.944131 19095 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 07:04:49.991293 19263 tablet.cc:2369] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:49.995759 19403 tablet.cc:2369] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:04:50.020573 19128 tablet.cc:2369] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:50.253876 19497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:50.254078 19497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:50.255067 19497 leader_election.cc:288] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64b3af9aa0f545aaa9ff16652a83a4fe (127.27.110.254:37665), ae51839e4c0a4cfabae9db793c894548 (127.27.110.253:44679)
I20260812 07:04:50.255678 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae51839e4c0a4cfabae9db793c894548" is_pre_election: true
I20260812 07:04:50.255694 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" is_pre_election: true
I20260812 07:04:50.255999 19300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b3a2bd0b2340819df6fccc592ea84f in term 0.
I20260812 07:04:50.256007 19168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2b3a2bd0b2340819df6fccc592ea84f in term 0.
I20260812 07:04:50.256464 19394 leader_election.cc:302] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b3af9aa0f545aaa9ff16652a83a4fe, b2b3a2bd0b2340819df6fccc592ea84f; no voters:
I20260812 07:04:50.256768 19497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:50.256865 19497 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:50.256915 19497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:50.257993 19497 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:50.258320 19497 leader_election.cc:288] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [CANDIDATE]: Term 1 election: Requested vote from peers 64b3af9aa0f545aaa9ff16652a83a4fe (127.27.110.254:37665), ae51839e4c0a4cfabae9db793c894548 (127.27.110.253:44679)
I20260812 07:04:50.258649 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae51839e4c0a4cfabae9db793c894548"
I20260812 07:04:50.258648 19168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe"
I20260812 07:04:50.258791 19300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:50.258790 19168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:50.260111 19300 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b3a2bd0b2340819df6fccc592ea84f in term 1.
I20260812 07:04:50.260111 19168 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2b3a2bd0b2340819df6fccc592ea84f in term 1.
I20260812 07:04:50.260411 19395 leader_election.cc:302] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae51839e4c0a4cfabae9db793c894548, b2b3a2bd0b2340819df6fccc592ea84f; no voters:
I20260812 07:04:50.260600 19497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:50.260811 19497 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [term 1 LEADER]: Becoming Leader. State: Replica: b2b3a2bd0b2340819df6fccc592ea84f, State: Running, Role: LEADER
I20260812 07:04:50.261134 19497 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } }
I20260812 07:04:50.263051 19574 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b3a2bd0b2340819df6fccc592ea84f. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.263209 19574 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:50.263587 19577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:50.264974 19577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:50.269712 19168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [term 1 FOLLOWER]: Refusing update from remote peer b2b3a2bd0b2340819df6fccc592ea84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:50.269712 19300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [term 1 FOLLOWER]: Refusing update from remote peer b2b3a2bd0b2340819df6fccc592ea84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:50.270427 19574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:50.270685 19497 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:50.274149 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b3a2bd0b2340819df6fccc592ea84f. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.274289 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:50.274609 19524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b3a2bd0b2340819df6fccc592ea84f. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.274742 19524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:50.277634 19497 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.277794 19497 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:50.277834 19546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.277946 19546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:50.277915 19497 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.277997 19497 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:50.278286 19584 mvcc.cc:205] Tried to move back new op lower bound from 7317578916937801728 to 7317578916912177152. Current Snapshot: MvccSnapshot[applied={T|T < 7317578916937801728}]
I20260812 07:04:50.279021 19524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b3af9aa0f545aaa9ff16652a83a4fe" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 37665 } } peers { permanent_uuid: "ae51839e4c0a4cfabae9db793c894548" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 44679 } } peers { permanent_uuid: "b2b3a2bd0b2340819df6fccc592ea84f" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 46547 } } }
I20260812 07:04:50.279152 19524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:04:50.280612 19577 catalog_manager.cc:1347] Generated new cluster ID: 1f205bad8cf9460da29252f8a3432181
I20260812 07:04:50.280682 19577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:50.340627 19577 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:50.341914 19577 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:50.359344 19577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2b3a2bd0b2340819df6fccc592ea84f: Generated new TSK 0
I20260812 07:04:50.360249 19577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:04:50.930089 19542 catalog_manager.cc:1259] Loaded cluster ID: 1f205bad8cf9460da29252f8a3432181
I20260812 07:04:50.930166 19542 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: loading cluster ID for follower catalog manager: success
I20260812 07:04:50.931967 19542 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: acquiring CA information for follower catalog manager: success
I20260812 07:04:50.933214 19542 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae51839e4c0a4cfabae9db793c894548: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:04:50.939452 19570 catalog_manager.cc:1259] Loaded cluster ID: 1f205bad8cf9460da29252f8a3432181
I20260812 07:04:50.939512 19570 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: loading cluster ID for follower catalog manager: success
I20260812 07:04:50.941071 19570 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: acquiring CA information for follower catalog manager: success
I20260812 07:04:50.942142 19570 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64b3af9aa0f545aaa9ff16652a83a4fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:04:51.441337 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 19095
I20260812 07:04:51.455168 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 19225
I20260812 07:04:51.468075 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 19360
2026-08-12T07:04:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3817 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:51 dist-test-slave-w4v5 krb5kdc[19615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:51 dist-test-slave-w4v5 krb5kdc[19615](info): set up 2 sockets
Aug 12 07:04:51 dist-test-slave-w4v5 krb5kdc[19615](info): commencing operation
krb5kdc: starting...
W20260812 07:04:52.776414 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:04:52 dist-test-slave-w4v5 krb5kdc[19615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:04:52.907392 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43403
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:53.023582 19633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:53.023856 19633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:53.023898 19633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:53.027776 19633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:53.027830 19633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:53.027849 19633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:53.027866 19633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:53.027881 19633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:53.033075 19633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:43403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:53.034116 19633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:53.035135 19633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:53.043793 19642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:53.043788 19644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:53.043849 19641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:53.044211 19633 server_base.cc:1034] running on GCE node
I20260812 07:04:53.044742 19633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:53.045801 19633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:53.046972 19633 hybrid_clock.cc:648] HybridClock initialized: now 1786518293046944 us; error 42 us; skew 500 ppm
Aug 12 07:04:53 dist-test-slave-w4v5 krb5kdc[19615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518293, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:53.049810 19633 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:53.050765 19633 webserver.cc:460] Webserver started at http://127.27.110.254:38547/ using document root <none> and password file <none>
I20260812 07:04:53.051420 19633 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:53.051513 19633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:53.051753 19633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:53.053791 19633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "0288ceeb7c344b588de4c41825994e0f"
format_stamp: "Formatted at 2026-08-12 07:04:53 on dist-test-slave-w4v5"
server_key: "4ee1e6d40f93cf75b42129d0485d4e2b"
server_key_iv: "1734126b4c345c10be6b8a9ba91a0502"
server_key_version: "encryptionkey@0"
I20260812 07:04:53.054371 19633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "0288ceeb7c344b588de4c41825994e0f"
format_stamp: "Formatted at 2026-08-12 07:04:53 on dist-test-slave-w4v5"
server_key: "4ee1e6d40f93cf75b42129d0485d4e2b"
server_key_iv: "1734126b4c345c10be6b8a9ba91a0502"
server_key_version: "encryptionkey@0"
I20260812 07:04:53.058297 19633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 07:04:53.060853 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:53.062068 19633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:04:53.062239 19633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "0288ceeb7c344b588de4c41825994e0f"
format_stamp: "Formatted at 2026-08-12 07:04:53 on dist-test-slave-w4v5"
server_key: "4ee1e6d40f93cf75b42129d0485d4e2b"
server_key_iv: "1734126b4c345c10be6b8a9ba91a0502"
server_key_version: "encryptionkey@0"
I20260812 07:04:53.062357 19633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:53.080998 19633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:53.084013 19633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:53.084227 19633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:53.091710 19633 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:43403
I20260812 07:04:53.091741 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:43403 every 8 connection(s)
I20260812 07:04:53.092686 19633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:53.095587 19737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:53.098059 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 19633
I20260812 07:04:53.098191 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:04:53.098487 28091 external_mini_cluster.cc:1428] Setting key 64cbccfe25b9e55f9e0b03fa62776401
I20260812 07:04:53.102099 19737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f: Bootstrap starting.
I20260812 07:04:53.104406 19737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:53.105199 19737 log.cc:824] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:53.107692 19737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f: No bootstrap required, opened a new log
Aug 12 07:04:53 dist-test-slave-w4v5 krb5kdc[19615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:53.111615 19737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0288ceeb7c344b588de4c41825994e0f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:04:53.111865 19737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:53.111968 19737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0288ceeb7c344b588de4c41825994e0f, State: Initialized, Role: FOLLOWER
I20260812 07:04:53.112493 19737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0288ceeb7c344b588de4c41825994e0f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:04:53.112674 19737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:53.112762 19737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:53.112864 19737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:53.113401 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:53.099924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.27.110.254:43403)
0812 07:04:53.100385 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:53.100401 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:53.100438 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:04:53.101226 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:53.101246 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:53.101579 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:53.101925 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:53.102773 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.103841 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.104763 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:53.105239 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:53.105432 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:53.108507 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:53.108512 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:53.108526 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:53.108553 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:53.110287 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:53.110787 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:53.110793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:53.110805 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:53.110928 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:53.111262 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:53.111266 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:53.111268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:53.111665 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:53.111887 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:53.112192 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 07:04:53.112495 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":145,"threads_started":1}
I20260812 07:04:53.113852 19737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0288ceeb7c344b588de4c41825994e0f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:04:53.114185 19737 leader_election.cc:302] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0288ceeb7c344b588de4c41825994e0f; no voters:
I20260812 07:04:53.114552 19737 leader_election.cc:288] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:53.114676 19745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:53.114925 19745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [term 1 LEADER]: Becoming Leader. State: Replica: 0288ceeb7c344b588de4c41825994e0f, State: Running, Role: LEADER
I20260812 07:04:53.115302 19745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0288ceeb7c344b588de4c41825994e0f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } }
I20260812 07:04:53.115857 19737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:53.116505 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:04:53.116645 19745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0288ceeb7c344b588de4c41825994e0f. Latest consensus state: current_term: 1 leader_uuid: "0288ceeb7c344b588de4c41825994e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0288ceeb7c344b588de4c41825994e0f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } }
I20260812 07:04:53.116852 19745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:53.117067 19749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0288ceeb7c344b588de4c41825994e0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0288ceeb7c344b588de4c41825994e0f" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43403 } } }
I20260812 07:04:53.117177 19749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:53.117439 19760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:53.120698 19760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:53.126353 19760 catalog_manager.cc:1347] Generated new cluster ID: dd73aa8e3d4a4cbba6de045f195ca070
I20260812 07:04:53.126437 19760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:53.136834 19760 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:53.138028 19760 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:53.146885 19760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0288ceeb7c344b588de4c41825994e0f: Generated new TSK 0
I20260812 07:04:53.147709 19760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:04:53.172724 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 19633
2026-08-12T07:04:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1707 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:04:53 dist-test-slave-w4v5 krb5kdc[19782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:04:53 dist-test-slave-w4v5 krb5kdc[19782](info): set up 2 sockets
Aug 12 07:04:53 dist-test-slave-w4v5 krb5kdc[19782](info): commencing operation
krb5kdc: starting...
W20260812 07:04:54.487735 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.282s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:04:54 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:04:54.614318 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39337
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:54.726336 19805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:54.726665 19805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:54.726742 19805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:54.730865 19805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:54.730957 19805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:54.730993 19805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:54.731032 19805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:54.731086 19805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:54.736546 19805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:54.737669 19805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:54.738741 19805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:54.747340 19827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:54.747496 19805 server_base.cc:1034] running on GCE node
W20260812 07:04:54.747368 19818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:54.747426 19815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:54.748142 19805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:54.749233 19805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:54.750422 19805 hybrid_clock.cc:648] HybridClock initialized: now 1786518294750405 us; error 29 us; skew 500 ppm
Aug 12 07:04:54 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518294, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:54.756152 19805 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:54.757318 19805 webserver.cc:460] Webserver started at http://127.27.110.254:45647/ using document root <none> and password file <none>
I20260812 07:04:54.758066 19805 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:54.758167 19805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:54.758424 19805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:54.761433 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "781082ca25c8457a8db49f550f28a84a"
format_stamp: "Formatted at 2026-08-12 07:04:54 on dist-test-slave-w4v5"
server_key: "07a75292c728b68c554bcfbd7a4fa74a"
server_key_iv: "12018efdd2d8c27b5f043be775200835"
server_key_version: "encryptionkey@0"
I20260812 07:04:54.762122 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "781082ca25c8457a8db49f550f28a84a"
format_stamp: "Formatted at 2026-08-12 07:04:54 on dist-test-slave-w4v5"
server_key: "07a75292c728b68c554bcfbd7a4fa74a"
server_key_iv: "12018efdd2d8c27b5f043be775200835"
server_key_version: "encryptionkey@0"
I20260812 07:04:54.766377 19805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:04:54.768725 19841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:54.769907 19805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:04:54.770044 19805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "781082ca25c8457a8db49f550f28a84a"
format_stamp: "Formatted at 2026-08-12 07:04:54 on dist-test-slave-w4v5"
server_key: "07a75292c728b68c554bcfbd7a4fa74a"
server_key_iv: "12018efdd2d8c27b5f043be775200835"
server_key_version: "encryptionkey@0"
I20260812 07:04:54.770149 19805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:54.784329 19805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:54.787202 19805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:54.787395 19805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:54.794502 19805 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39337
I20260812 07:04:54.794780 19936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39337 every 8 connection(s)
I20260812 07:04:54.795495 19805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:54.798146 19940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:54.800916 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 19805
I20260812 07:04:54.801040 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:04:54.801355 28091 external_mini_cluster.cc:1428] Setting key 2d8d78b8ed029ca67f61e59750658d60
I20260812 07:04:54.804129 19940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap starting.
I20260812 07:04:54.806648 19940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:54.807559 19940 log.cc:824] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:54.809983 19940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: No bootstrap required, opened a new log
Aug 12 07:04:54 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:54.813230 19940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:54.813505 19940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:54.813587 19940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781082ca25c8457a8db49f550f28a84a, State: Initialized, Role: FOLLOWER
I20260812 07:04:54.814074 19940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:54.814230 19940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:54.814316 19940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:54.814420 19940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:54.815460 19940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:54.815810 19940 leader_election.cc:302] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781082ca25c8457a8db49f550f28a84a; no voters:
I20260812 07:04:54.816174 19940 leader_election.cc:288] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:04:54.816248 19947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:54.816478 19947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 LEADER]: Becoming Leader. State: Replica: 781082ca25c8457a8db49f550f28a84a, State: Running, Role: LEADER
I20260812 07:04:54.816845 19947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:54.817195 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:54.802658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35600 (local address 127.27.110.254:39337)
0812 07:04:54.803047 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:54.803114 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:54.803169 (+ 55us) server_negotiation.cc:407] Connection header received
0812 07:04:54.803770 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:54.803788 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:54.804100 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:54.804479 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:54.805326 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.806355 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.807589 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:54.808214 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:54.808390 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:54.811408 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:54.811413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:54.811427 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:54.811455 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:54.813862 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:54.814344 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:54.814350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:54.814357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:54.814533 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:54.814815 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:54.814819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:54.814821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:54.815311 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:54.815529 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:54.815821 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 07:04:54.816159 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":129,"threads_started":1}
I20260812 07:04:54.817548 19940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:54.818569 19950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781082ca25c8457a8db49f550f28a84a. Latest consensus state: current_term: 1 leader_uuid: "781082ca25c8457a8db49f550f28a84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } } }
I20260812 07:04:54.818717 19950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:54.819129 19948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "781082ca25c8457a8db49f550f28a84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } } }
I20260812 07:04:54.819226 19948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:54.819445 19963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:54.822609 19963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:54.829064 19963 catalog_manager.cc:1347] Generated new cluster ID: 30c5e06cf18842a59b7b96af20826814
I20260812 07:04:54.829172 19963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:54.839771 19963 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:04:54.840821 19963 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:54.854072 19963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Generated new TSK 0
I20260812 07:04:54.854753 19963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:04:54.888258 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39337
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:55.004145 19983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:55.004477 19983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:55.004568 19983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:55.008481 19983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:55.008562 19983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:55.008652 19983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:04:55.013944 19983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:55.015091 19983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:55.016137 19983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:55.026111 19994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:55.026273 19983 server_base.cc:1034] running on GCE node
W20260812 07:04:55.026142 19999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:55.026151 19995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:55.026737 19983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:55.027436 19983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:55.028632 19983 hybrid_clock.cc:648] HybridClock initialized: now 1786518295028615 us; error 29 us; skew 500 ppm
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:55.031733 19983 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:04:55.032627 19983 webserver.cc:460] Webserver started at http://127.27.110.193:38617/ using document root <none> and password file <none>
I20260812 07:04:55.033200 19983 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:55.033264 19983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:55.033473 19983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:55.035718 19983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "dfa773df8ace4664816635a558cff9ed"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "218b6494137171ad7c32b80fb620b6cc"
server_key_iv: "25aa5e2d07c7c505632b3fc06fe31dd3"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.036279 19983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "dfa773df8ace4664816635a558cff9ed"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "218b6494137171ad7c32b80fb620b6cc"
server_key_iv: "25aa5e2d07c7c505632b3fc06fe31dd3"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.040113 19983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:04:55.042565 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.043848 19983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:04:55.043952 19983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "dfa773df8ace4664816635a558cff9ed"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "218b6494137171ad7c32b80fb620b6cc"
server_key_iv: "25aa5e2d07c7c505632b3fc06fe31dd3"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.044083 19983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:55.066344 19983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:55.069599 19983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:55.069796 19983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:55.070392 19983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:55.071270 19983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:55.071321 19983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.071385 19983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:55.071417 19983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.080936 19983 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:37731
I20260812 07:04:55.080963 20219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:37731 every 8 connection(s)
I20260812 07:04:55.081874 19983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:55.085765 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 19983
I20260812 07:04:55.085892 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:04:55.086227 28091 external_mini_cluster.cc:1428] Setting key 0ba14ebe395b5b87561892259c0a9ce6
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:04:55.097520 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.083702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:59025 (local address 127.27.110.254:39337)
0812 07:04:55.083834 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.083838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.084701 (+ 863us) server_negotiation.cc:407] Connection header received
0812 07:04:55.086319 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.086323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.086382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.086459 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.087737 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.088265 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.090071 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.090471 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.090602 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.093582 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:55.093585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:55.093587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:55.093611 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:55.095211 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.095863 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.095866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.095867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.095947 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.096354 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.096357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.096358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.096528 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:55.096604 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.097233 (+ 629us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.097387 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:04:55.098472 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.084088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:59025)
0812 07:04:55.084549 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.084592 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.086076 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.086678 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.086687 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.087041 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:55.087576 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.087590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.088403 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.088410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.089921 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.089935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.090649 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.090652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.090822 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.091451 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:55.091467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:55.093376 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:55.095328 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:55.095337 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:55.095353 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:55.095761 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:55.096056 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:55.096061 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:55.096063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:55.096256 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:55.096646 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:55.096655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:55.096992 (+ 337us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.097206 (+ 214us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.097587 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:04:55.099839 20221 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39337
I20260812 07:04:55.100104 20221 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:55.100703 20221 heartbeater.cc:511] Master 127.27.110.254:39337 requested a full tablet report, sending...
I20260812 07:04:55.102281 19877 ts_manager.cc:194] Registered new tserver with Master: dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731)
I20260812 07:04:55.103482 19877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:59025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:04:55.117666 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39337
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:55.231825 20227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:55.232142 20227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:55.232245 20227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:55.236286 20227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:55.236368 20227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:55.236455 20227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:04:55.241776 20227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:55.242960 20227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:55.244071 20227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:55.254504 20239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:55.254525 20232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:55.254608 20235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:55.254837 20227 server_base.cc:1034] running on GCE node
I20260812 07:04:55.255335 20227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:55.255950 20227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:55.257138 20227 hybrid_clock.cc:648] HybridClock initialized: now 1786518295257094 us; error 46 us; skew 500 ppm
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:55.260483 20227 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:04:55.261390 20227 webserver.cc:460] Webserver started at http://127.27.110.194:33813/ using document root <none> and password file <none>
I20260812 07:04:55.261956 20227 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:55.262018 20227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:55.262281 20227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:55.264418 20227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "c6b72a0e83ff4d5b8857601bf358d515"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "828cc4abb72281d31e5f6fc8823003ca"
server_key_iv: "30634e3d8147685a0d239fb6973ad564"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.264971 20227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b72a0e83ff4d5b8857601bf358d515"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "828cc4abb72281d31e5f6fc8823003ca"
server_key_iv: "30634e3d8147685a0d239fb6973ad564"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.268788 20227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:04:55.271380 20249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.273121 20227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:04:55.273253 20227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "c6b72a0e83ff4d5b8857601bf358d515"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "828cc4abb72281d31e5f6fc8823003ca"
server_key_iv: "30634e3d8147685a0d239fb6973ad564"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.273366 20227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:55.289122 20227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:55.292999 20227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:55.293228 20227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:55.294005 20227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:55.294837 20227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:55.294884 20227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.294924 20227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:55.294940 20227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.304241 20227 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:38777
I20260812 07:04:55.304250 20437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:38777 every 8 connection(s)
I20260812 07:04:55.305181 20227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:55.314456 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 20227
I20260812 07:04:55.314589 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:04:55.314913 28091 external_mini_cluster.cc:1428] Setting key a8a6ee819d08abf9347545e2a81a29e0
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:55.319577 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.306990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:55159 (local address 127.27.110.254:39337)
0812 07:04:55.307169 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.307173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.308022 (+ 849us) server_negotiation.cc:407] Connection header received
0812 07:04:55.309495 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.309500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.309560 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.309641 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.310795 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.311389 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.312674 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.312900 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.313011 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.315618 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:55.315621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:55.315624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:55.315649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:55.317177 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.317835 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.317838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.317839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.317920 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.318329 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.318331 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.318333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.318504 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:55.318585 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.319267 (+ 682us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.319398 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:04:55.320426 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.307409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:55159)
0812 07:04:55.307873 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.307915 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.309283 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.309803 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.309809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.310165 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:55.310655 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.310667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.311517 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.311520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.312538 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.312555 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.313051 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.313055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.313229 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.313768 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:55.313784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:55.315419 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:55.317292 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:55.317301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:55.317318 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:55.317739 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:55.318013 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:55.318019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:55.318023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:55.318215 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:55.318630 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:55.318639 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:55.318979 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.319200 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.319526 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":130,"threads_started":1}
I20260812 07:04:55.321755 20440 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39337
I20260812 07:04:55.322016 20440 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:55.322571 20440 heartbeater.cc:511] Master 127.27.110.254:39337 requested a full tablet report, sending...
I20260812 07:04:55.323709 19877 ts_manager.cc:194] Registered new tserver with Master: c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777)
I20260812 07:04:55.324611 19877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:55159
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:04:55.346436 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39337
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:55.460351 20446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:55.460597 20446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:55.460675 20446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:55.464617 20446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:55.464670 20446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:55.464764 20446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:04:55.469928 20446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:55.471035 20446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:55.472085 20446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:55.481813 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:55.481895 20446 server_base.cc:1034] running on GCE node
W20260812 07:04:55.481817 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:55.481793 20457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:55.482439 20446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:55.483011 20446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:55.484210 20446 hybrid_clock.cc:648] HybridClock initialized: now 1786518295484197 us; error 25 us; skew 500 ppm
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:55.486892 20446 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:04:55.487828 20446 webserver.cc:460] Webserver started at http://127.27.110.195:38863/ using document root <none> and password file <none>
I20260812 07:04:55.488386 20446 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:55.488447 20446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:55.488670 20446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:04:55.490665 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "fa082d3e7a21412fa5276bfefd023f19"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "4230409bd9b39f7ab0e8bc7088bc85c5"
server_key_iv: "b24755459126741614f3b24f8c2306e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.491274 20446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "fa082d3e7a21412fa5276bfefd023f19"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "4230409bd9b39f7ab0e8bc7088bc85c5"
server_key_iv: "b24755459126741614f3b24f8c2306e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.494916 20446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:55.497298 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.498461 20446 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:55.498581 20446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "fa082d3e7a21412fa5276bfefd023f19"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "4230409bd9b39f7ab0e8bc7088bc85c5"
server_key_iv: "b24755459126741614f3b24f8c2306e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.498688 20446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:55.524770 20446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:55.528337 20446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:55.528587 20446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:55.529254 20446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:55.530104 20446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:55.530154 20446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.530242 20446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:55.530279 20446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.539448 20446 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44517
I20260812 07:04:55.539484 20665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44517 every 8 connection(s)
I20260812 07:04:55.540364 20446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:55.543541 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 20446
I20260812 07:04:55.543663 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:04:55.543941 28091 external_mini_cluster.cc:1428] Setting key 681a6ab1f399b5509ac2965aa296afef
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:55.554309 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.542179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:50097 (local address 127.27.110.254:39337)
0812 07:04:55.542315 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.542319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.543257 (+ 938us) server_negotiation.cc:407] Connection header received
0812 07:04:55.544841 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.544845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.544903 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.544991 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.546176 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.546755 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.548162 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.548395 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.548504 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.550926 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:55.550930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:55.550933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:55.550963 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:55.552292 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.552790 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.552794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.552796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.552882 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.553226 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.553229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.553232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.553419 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:55.553522 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.554020 (+ 498us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.554143 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 07:04:55.555178 20668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.542612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:50097)
0812 07:04:55.543060 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.543139 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.544622 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.545177 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.545184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.545545 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:55.546027 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.546040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.546891 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.546894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.548038 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.548048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.548547 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.548551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.548773 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.549287 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:55.549303 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:55.550764 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:55.552408 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:55.552414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:55.552428 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:55.552686 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:55.552974 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:55.552977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:55.552979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:55.553118 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:55.553536 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:55.553545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:55.553790 (+ 245us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.553978 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.554309 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260812 07:04:55.556264 20666 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39337
I20260812 07:04:55.556517 20666 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:55.557067 20666 heartbeater.cc:511] Master 127.27.110.254:39337 requested a full tablet report, sending...
I20260812 07:04:55.558068 19877 ts_manager.cc:194] Registered new tserver with Master: fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
I20260812 07:04:55.558648 19877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:50097
I20260812 07:04:55.569365 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:04:55.578788 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.571362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34726 (local address 127.27.110.254:39337)
0812 07:04:55.571507 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.571513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.571661 (+ 148us) server_negotiation.cc:407] Connection header received
0812 07:04:55.571760 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.571763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.571801 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.571866 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:55.572546 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.573067 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.573974 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.574186 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.574261 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.575613 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:55.575616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:55.575618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:55.575640 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:55.577117 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.577492 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.577495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.577496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.577568 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:55.577893 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:55.577898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:55.577900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:55.578157 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:55.578263 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.578499 (+ 236us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.578619 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:04:55.581602 19877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:55.584694 19877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:55.599195 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.593921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.27.110.195:44517)
0812 07:04:55.594613 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.594628 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.594652 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:04:55.594731 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.594735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.594917 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.595068 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:55.595281 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.596134 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.597556 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.598396 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.598545 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.598687 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.598873 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.598984 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":171,"threads_started":1}
I20260812 07:04:55.599318 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.594037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:38777 (local address 127.0.0.1:33480)
0812 07:04:55.594414 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.594431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.594508 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.595376 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.595379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.595396 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:55.595472 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.595477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.596786 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.596790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.597868 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.597876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.598907 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.598911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.599032 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.599053 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.599143 (+ 90us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.599201 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":25,"thread_start_us":94,"threads_started":1}
I20260812 07:04:55.599660 20689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.593731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44517 (local address 127.0.0.1:53642)
0812 07:04:55.594275 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.594289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.594370 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.595045 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.595048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.595066 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:55.595183 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.595190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.596420 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.596425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.597436 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.597446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.598533 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.598537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.598670 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.598718 (+ 48us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.599007 (+ 289us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.599125 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":40,"thread_start_us":90,"threads_started":1}
I20260812 07:04:55.599795 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.594467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.27.110.194:38777)
0812 07:04:55.594890 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.594902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.594923 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:04:55.594994 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.595000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.595222 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.595399 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:55.595557 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.596626 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.597987 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.598593 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.598714 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.598841 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.599524 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.599632 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":194,"threads_started":1}
I20260812 07:04:55.599318 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.592878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:37731 (local address 127.0.0.1:47392)
0812 07:04:55.593221 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.593269 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.593420 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.594092 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.594097 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.594118 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:55.594273 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.594281 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.596160 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.596163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.597559 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.597570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.598752 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.598756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.598859 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.598883 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.598954 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.599079 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":26,"thread_start_us":106,"threads_started":1}
I20260812 07:04:55.600239 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.593011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.27.110.193:37731)
0812 07:04:55.593273 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.593292 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.593379 (+ 87us) server_negotiation.cc:407] Connection header received
0812 07:04:55.593604 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.593611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.593847 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.594605 (+ 758us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:55.594801 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.596001 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.597699 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.598466 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.598572 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.598687 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.599212 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.600103 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260812 07:04:55.602608 20564 tablet_service.cc:1467] Processing CreateTablet for tablet c7d485ca153d44128aca918b986bd9de (DEFAULT_TABLE table=test-table [id=f93311083bf649d4a30cccbbd0e0d77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:55.602699 20342 tablet_service.cc:1467] Processing CreateTablet for tablet c7d485ca153d44128aca918b986bd9de (DEFAULT_TABLE table=test-table [id=f93311083bf649d4a30cccbbd0e0d77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:55.602909 20107 tablet_service.cc:1467] Processing CreateTablet for tablet c7d485ca153d44128aca918b986bd9de (DEFAULT_TABLE table=test-table [id=f93311083bf649d4a30cccbbd0e0d77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:55.603701 20564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d485ca153d44128aca918b986bd9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:55.603780 20342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d485ca153d44128aca918b986bd9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:55.603780 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d485ca153d44128aca918b986bd9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:55.609735 20691 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Bootstrap starting.
I20260812 07:04:55.611271 20692 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Bootstrap starting.
I20260812 07:04:55.611271 20695 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Bootstrap starting.
I20260812 07:04:55.612658 20691 tablet_bootstrap.cc:654] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:55.613389 20691 log.cc:824] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:55.613422 20692 tablet_bootstrap.cc:654] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:55.613910 20695 tablet_bootstrap.cc:654] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:55.614348 20692 log.cc:824] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:55.614766 20695 log.cc:824] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:55.615569 20691 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: No bootstrap required, opened a new log
I20260812 07:04:55.615789 20691 ts_tablet_manager.cc:1397] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:04:55.616446 20692 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: No bootstrap required, opened a new log
I20260812 07:04:55.616643 20692 ts_tablet_manager.cc:1397] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:04:55.616990 20695 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: No bootstrap required, opened a new log
I20260812 07:04:55.617180 20695 ts_tablet_manager.cc:1397] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:04:55.619549 20691 raft_consensus.cc:348] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.619829 20691 raft_consensus.cc:374] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:55.619894 20691 raft_consensus.cc:729] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa082d3e7a21412fa5276bfefd023f19, State: Initialized, Role: FOLLOWER
I20260812 07:04:55.620330 20691 consensus_queue.cc:260] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.620981 20666 heartbeater.cc:503] Master 127.27.110.254:39337 was elected leader, sending a full tablet report...
I20260812 07:04:55.621284 20691 ts_tablet_manager.cc:1428] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:04:55.621233 20695 raft_consensus.cc:348] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.621553 20695 raft_consensus.cc:374] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:55.621620 20695 raft_consensus.cc:729] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b72a0e83ff4d5b8857601bf358d515, State: Initialized, Role: FOLLOWER
I20260812 07:04:55.621775 20692 raft_consensus.cc:348] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.622028 20692 raft_consensus.cc:374] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:55.622079 20692 raft_consensus.cc:729] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa773df8ace4664816635a558cff9ed, State: Initialized, Role: FOLLOWER
I20260812 07:04:55.622052 20695 consensus_queue.cc:260] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.622488 20692 consensus_queue.cc:260] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.622665 20440 heartbeater.cc:503] Master 127.27.110.254:39337 was elected leader, sending a full tablet report...
I20260812 07:04:55.623234 20695 ts_tablet_manager.cc:1428] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 07:04:55.623462 20692 ts_tablet_manager.cc:1428] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260812 07:04:55.623903 20221 heartbeater.cc:503] Master 127.27.110.254:39337 was elected leader, sending a full tablet report...
I20260812 07:04:55.642800 20703 raft_consensus.cc:482] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:55.643045 20703 raft_consensus.cc:504] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.644357 20703 leader_election.cc:288] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517), c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777)
I20260812 07:04:55.648727 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.644829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:40701 (local address 127.27.110.195:44517)
0812 07:04:55.645005 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.645009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.645215 (+ 206us) server_negotiation.cc:407] Connection header received
0812 07:04:55.645385 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.645389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.645446 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.645505 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:55.645903 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.646415 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.647711 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.648160 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.648286 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.648340 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.648522 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.648618 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:04:55.648761 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.644731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44517 (local address 127.27.110.193:40701)
0812 07:04:55.645100 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.645119 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.645236 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.645636 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.645638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.645667 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:55.645783 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.645787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.646525 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.646531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.647594 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.647605 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.648286 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.648290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.648400 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.648419 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.648513 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.648587 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":37,"spinlock_wait_cycles":3328,"thread_start_us":168,"threads_started":1}
I20260812 07:04:55.648922 20690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.645375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:40455 (local address 127.27.110.194:38777)
0812 07:04:55.645509 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.645513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.645636 (+ 123us) server_negotiation.cc:407] Connection header received
0812 07:04:55.645799 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.645802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.645848 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.645938 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:55.646226 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.646759 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.647935 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.648405 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.648525 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.648580 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.648746 (+ 166us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.648812 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 07:04:55.648940 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.645208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:38777 (local address 127.27.110.193:40455)
0812 07:04:55.645531 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.645547 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:55.645694 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:55.645949 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:55.645952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:55.645968 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:55.646111 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.646116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.646892 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.646895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.647813 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:55.647824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.648510 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.648514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:55.648629 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.648658 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 07:04:55.648730 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 07:04:55.648802 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260812 07:04:55.649505 20374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d485ca153d44128aca918b986bd9de" candidate_uuid: "dfa773df8ace4664816635a558cff9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b72a0e83ff4d5b8857601bf358d515" is_pre_election: true
I20260812 07:04:55.649525 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d485ca153d44128aca918b986bd9de" candidate_uuid: "dfa773df8ace4664816635a558cff9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa082d3e7a21412fa5276bfefd023f19" is_pre_election: true
I20260812 07:04:55.649840 20374 raft_consensus.cc:2454] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa773df8ace4664816635a558cff9ed in term 0.
I20260812 07:04:55.649840 20592 raft_consensus.cc:2454] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfa773df8ace4664816635a558cff9ed in term 0.
I20260812 07:04:55.650388 20012 leader_election.cc:302] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b72a0e83ff4d5b8857601bf358d515, dfa773df8ace4664816635a558cff9ed; no voters:
I20260812 07:04:55.650686 20703 raft_consensus.cc:2781] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:55.651115 20703 raft_consensus.cc:482] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:55.651176 20703 raft_consensus.cc:3037] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:55.652189 20703 raft_consensus.cc:504] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.652526 20703 leader_election.cc:288] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 election: Requested vote from peers fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517), c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777)
I20260812 07:04:55.652843 20592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d485ca153d44128aca918b986bd9de" candidate_uuid: "dfa773df8ace4664816635a558cff9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa082d3e7a21412fa5276bfefd023f19"
I20260812 07:04:55.652884 20374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7d485ca153d44128aca918b986bd9de" candidate_uuid: "dfa773df8ace4664816635a558cff9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b72a0e83ff4d5b8857601bf358d515"
I20260812 07:04:55.652977 20592 raft_consensus.cc:3037] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:55.652995 20374 raft_consensus.cc:3037] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:55.654035 20374 raft_consensus.cc:2454] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa773df8ace4664816635a558cff9ed in term 1.
I20260812 07:04:55.654053 20592 raft_consensus.cc:2454] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfa773df8ace4664816635a558cff9ed in term 1.
I20260812 07:04:55.654292 20012 leader_election.cc:302] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b72a0e83ff4d5b8857601bf358d515, dfa773df8ace4664816635a558cff9ed; no voters:
I20260812 07:04:55.654491 20703 raft_consensus.cc:2781] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:55.654758 20703 raft_consensus.cc:686] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 LEADER]: Becoming Leader. State: Replica: dfa773df8ace4664816635a558cff9ed, State: Running, Role: LEADER
I20260812 07:04:55.655114 20703 consensus_queue.cc:237] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:55.658314 19873 catalog_manager.cc:5495] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed reported cstate change: term changed from 0 to 1, leader changed from <none> to dfa773df8ace4664816635a558cff9ed (127.27.110.193). New cstate: current_term: 1 leader_uuid: "dfa773df8ace4664816635a558cff9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:55.680455 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.676518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.27.110.193:37731)
0812 07:04:55.676668 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:55.676673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:55.676688 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:04:55.676760 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:55.676762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:55.676815 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:55.676917 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:55.677188 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.677703 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.678998 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:55.679294 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:55.679377 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:55.679671 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:55.680065 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:55.680159 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:55.680219 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 07:04:55.680272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:04:55.686149 20374 raft_consensus.cc:1261] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Refusing update from remote peer dfa773df8ace4664816635a558cff9ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:55.686177 20592 raft_consensus.cc:1261] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Refusing update from remote peer dfa773df8ace4664816635a558cff9ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:55.686861 20703 consensus_queue.cc:1035] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:55.687054 20710 consensus_queue.cc:1035] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:55.697245 20720 mvcc.cc:205] Tried to move back new op lower bound from 7317578939123585024 to 7317578939006345216. Current Snapshot: MvccSnapshot[applied={T|T < 7317578939123585024}]
I20260812 07:04:55.702907 19873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:55.703109 19873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:55.705384 19873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Sending DeleteTablet for 3 replicas of tablet c7d485ca153d44128aca918b986bd9de
I20260812 07:04:55.706188 20564 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d485ca153d44128aca918b986bd9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:55 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:53642
I20260812 07:04:55.706274 20342 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d485ca153d44128aca918b986bd9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:55 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:33480
I20260812 07:04:55.706270 20107 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d485ca153d44128aca918b986bd9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:55 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:47392
I20260812 07:04:55.706822 20738 tablet_replica.cc:330] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: stopping tablet replica
I20260812 07:04:55.706871 20741 tablet_replica.cc:330] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: stopping tablet replica
I20260812 07:04:55.706907 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 19983
I20260812 07:04:55.710227 20742 tablet_replica.cc:330] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: stopping tablet replica
I20260812 07:04:55.710608 20738 raft_consensus.cc:2229] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:55.710675 20742 raft_consensus.cc:2229] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:55.710994 20738 raft_consensus.cc:2258] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:55.711128 20742 raft_consensus.cc:2258] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:55.714262 20738 ts_tablet_manager.cc:1905] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:55.717401 20738 ts_tablet_manager.cc:1918] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:55.717555 20738 log.cc:1196] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/c7d485ca153d44128aca918b986bd9de
I20260812 07:04:55.717939 20738 ts_tablet_manager.cc:1939] T c7d485ca153d44128aca918b986bd9de P fa082d3e7a21412fa5276bfefd023f19: Deleting consensus metadata
I20260812 07:04:55.719969 20742 ts_tablet_manager.cc:1905] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:55.720095 19842 catalog_manager.cc:4841] TS fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): tablet c7d485ca153d44128aca918b986bd9de (table test-table [id=f93311083bf649d4a30cccbbd0e0d77a]) successfully deleted
I20260812 07:04:55.723680 20742 ts_tablet_manager.cc:1918] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:55.723802 20742 log.cc:1196] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/c7d485ca153d44128aca918b986bd9de
I20260812 07:04:55.724222 20742 ts_tablet_manager.cc:1939] T c7d485ca153d44128aca918b986bd9de P c6b72a0e83ff4d5b8857601bf358d515: Deleting consensus metadata
I20260812 07:04:55.725698 19842 catalog_manager.cc:4841] TS c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): tablet c7d485ca153d44128aca918b986bd9de (table test-table [id=f93311083bf649d4a30cccbbd0e0d77a]) successfully deleted
I20260812 07:04:55.727010 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 20227
W20260812 07:04:55.726979 19842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:37731 (error 108)
I20260812 07:04:55.738863 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:55.734806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:37731 (local address 127.0.0.1:47418)
0812 07:04:55.738614 (+ 3808us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:55.738725 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:37731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3670}
W20260812 07:04:55.739379 19842 catalog_manager.cc:4584] TS dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7d485ca153d44128aca918b986bd9de: Network error: Client connection negotiation failed: client connection to 127.27.110.193:37731: connect: Connection refused (error 111)
I20260812 07:04:55.747493 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 20446
I20260812 07:04:55.756358 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 19805
I20260812 07:04:55.768253 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39337
--webserver_interface=127.27.110.254
--webserver_port=45647
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:55.887501 20744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:55.887815 20744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:55.887887 20744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:55.891873 20744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:55.891957 20744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:55.891996 20744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:55.892047 20744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:55.892118 20744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:55.897322 20744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:55.898402 20744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:55.899447 20744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:04:55.913784 20744 server_base.cc:1034] running on GCE node
W20260812 07:04:55.913686 20754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:55.913679 20753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:55.913689 20759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:55.914510 20744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:55.915751 20744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:55.916911 20744 hybrid_clock.cc:648] HybridClock initialized: now 1786518295916877 us; error 46 us; skew 500 ppm
Aug 12 07:04:55 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:55.919719 20744 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:55.920632 20744 webserver.cc:460] Webserver started at http://127.27.110.254:45647/ using document root <none> and password file <none>
I20260812 07:04:55.921216 20744 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:55.921325 20744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:55.925205 20744 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260812 07:04:55.927414 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:55.928630 20744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260812 07:04:55.928769 20744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "781082ca25c8457a8db49f550f28a84a"
format_stamp: "Formatted at 2026-08-12 07:04:54 on dist-test-slave-w4v5"
server_key: "07a75292c728b68c554bcfbd7a4fa74a"
server_key_iv: "12018efdd2d8c27b5f043be775200835"
server_key_version: "encryptionkey@0"
I20260812 07:04:55.929366 20744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:55.943753 20744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:55.946723 20744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:55.946925 20744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:55.954443 20744 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39337
I20260812 07:04:55.954757 20869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39337 every 8 connection(s)
I20260812 07:04:55.955639 20744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:55.957227 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 20744
I20260812 07:04:55.957710 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:37731
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:39337
--webserver_port=38617
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:55.960916 20870 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:04:55.962661 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap starting.
I20260812 07:04:55.980628 20870 log.cc:824] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:55.989499 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:55.989892 20870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap complete.
I20260812 07:04:55.994186 20870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:55.994577 20870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781082ca25c8457a8db49f550f28a84a, State: Initialized, Role: FOLLOWER
I20260812 07:04:55.995046 20870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:55.995210 20870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:55.995304 20870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:55.995414 20870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:04:55.997277 20870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:55.997606 20870 leader_election.cc:302] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781082ca25c8457a8db49f550f28a84a; no voters:
I20260812 07:04:55.997926 20870 leader_election.cc:288] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:04:55.998027 20881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:04:55.998251 20881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 LEADER]: Becoming Leader. State: Replica: 781082ca25c8457a8db49f550f28a84a, State: Running, Role: LEADER
I20260812 07:04:55.998605 20881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:55.999181 20870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:56.000087 20884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781082ca25c8457a8db49f550f28a84a. Latest consensus state: current_term: 2 leader_uuid: "781082ca25c8457a8db49f550f28a84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } } }
I20260812 07:04:56.000216 20884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:56.000527 20882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "781082ca25c8457a8db49f550f28a84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } } }
I20260812 07:04:56.000628 20882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:56.004022 20909 catalog_manager.cc:1259] Loaded cluster ID: 30c5e06cf18842a59b7b96af20826814
I20260812 07:04:56.004093 20909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: loading cluster ID for follower catalog manager: success
W20260812 07:04:56.004927 20909 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260812 07:04:56.005009 20909 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260812 07:04:56.005216 20896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:56.007241 20896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:56.007680 20896 catalog_manager.cc:1259] Loaded cluster ID: 30c5e06cf18842a59b7b96af20826814
I20260812 07:04:56.007756 20896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:04:56.008616 20896 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 07:04:56.011648 20896 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 07:04:56.016286 20896 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:04:56.016928 20896 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Loaded TSK: 0
I20260812 07:04:56.017876 20896 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:04:56.089017 20875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:56.089298 20875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:56.089370 20875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:56.093355 20875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:56.093431 20875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:56.093514 20875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:04:56.098693 20875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:37731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=38617
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:56.099920 20875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:56.101090 20875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:56.112304 20921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:56.112298 20927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:56.112308 20922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:56.112785 20875 server_base.cc:1034] running on GCE node
I20260812 07:04:56.113164 20875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:56.113852 20875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:56.115029 20875 hybrid_clock.cc:648] HybridClock initialized: now 1786518296115010 us; error 39 us; skew 500 ppm
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:56.117795 20875 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:04:56.118707 20875 webserver.cc:460] Webserver started at http://127.27.110.193:38617/ using document root <none> and password file <none>
I20260812 07:04:56.119302 20875 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:56.119395 20875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:56.123327 20875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:56.125571 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.126890 20875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:56.127019 20875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "dfa773df8ace4664816635a558cff9ed"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "218b6494137171ad7c32b80fb620b6cc"
server_key_iv: "25aa5e2d07c7c505632b3fc06fe31dd3"
server_key_version: "encryptionkey@0"
I20260812 07:04:56.127597 20875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:56.138425 20875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:56.141507 20875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:56.141734 20875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:56.142331 20875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:56.143534 20949 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 07:04:56.149195 20875 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 07:04:56.149273 20875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260812 07:04:56.149354 20875 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 07:04:56.151005 20875 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 07:04:56.151072 20875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 07:04:56.152554 20949 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Bootstrap starting.
I20260812 07:04:56.162319 20875 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:37731
I20260812 07:04:56.163432 20875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:56.163641 21154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:37731 every 8 connection(s)
I20260812 07:04:56.165771 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 20875
I20260812 07:04:56.166273 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:38777
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:39337
--webserver_port=33813
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:56.174108 20949 log.cc:824] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:56.182706 20949 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:56.183211 20949 tablet_bootstrap.cc:492] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Bootstrap complete.
I20260812 07:04:56.183665 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.165651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:50599 (local address 127.27.110.254:39337)
0812 07:04:56.167013 (+ 1362us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.167035 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.167106 (+ 71us) server_negotiation.cc:407] Connection header received
0812 07:04:56.168109 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.168128 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.168469 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.168741 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.170395 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.171497 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.173032 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.173714 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.173852 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.177475 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:56.177482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:56.177497 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:56.177531 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:56.179814 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.180575 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.180584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.180590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.180769 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.181178 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.181182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.181184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.181712 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:56.181876 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.182464 (+ 588us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.182767 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":130,"threads_started":1}
I20260812 07:04:56.183701 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.165635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:50599)
0812 07:04:56.166318 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.166381 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.167623 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.168883 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.168892 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.169373 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.170188 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.170217 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.171663 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.171667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.172876 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.172892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.174094 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.174098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.174331 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.175030 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.175060 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.177234 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:56.179952 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.179961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.179978 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.180395 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.180887 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.180890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.180892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.181063 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.181850 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:56.181856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:56.182117 (+ 261us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.182323 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.182640 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":152,"threads_started":1}
I20260812 07:04:56.185007 20949 ts_tablet_manager.cc:1397] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Time spent bootstrapping tablet: real 0.034s user 0.018s sys 0.012s
I20260812 07:04:56.186421 21156 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39337
I20260812 07:04:56.186650 21156 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:56.187216 21156 heartbeater.cc:511] Master 127.27.110.254:39337 requested a full tablet report, sending...
I20260812 07:04:56.187620 20949 raft_consensus.cc:348] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:56.188133 20949 raft_consensus.cc:729] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa773df8ace4664816635a558cff9ed, State: Initialized, Role: FOLLOWER
I20260812 07:04:56.188596 20949 consensus_queue.cc:260] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } }
I20260812 07:04:56.189523 20949 ts_tablet_manager.cc:1428] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260812 07:04:56.189888 20797 ts_manager.cc:194] Registered new tserver with Master: dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731)
I20260812 07:04:56.193746 20797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:50599
I20260812 07:04:56.196074 21156 heartbeater.cc:503] Master 127.27.110.254:39337 was elected leader, sending a full tablet report...
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518295, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:04:56.204053 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.193110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.27.110.193:37731)
0812 07:04:56.193480 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.193491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.193839 (+ 348us) server_negotiation.cc:407] Connection header received
0812 07:04:56.194037 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.194047 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.194196 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.194335 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.195045 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.196184 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.197198 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.197582 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.197700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.200282 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:56.200293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:56.200306 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:56.200353 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:56.202122 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.202653 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.202659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.202661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.202738 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.203042 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.203044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.203045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.203318 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:56.203418 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.203813 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.203939 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":100,"threads_started":1}
I20260812 07:04:56.204070 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.193073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:37731 (local address 127.0.0.1:47434)
0812 07:04:56.193720 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.193752 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.193903 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.194337 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.194342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.194772 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.194910 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.194918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.196329 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.196333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.197076 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.197085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.197851 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.197854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.197931 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.198464 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.198478 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.200123 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:56.202236 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.202246 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.202251 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.202548 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.202828 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.202833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.202836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.202939 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.203431 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:56.203437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:56.203562 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.203714 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.203898 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":119,"threads_started":1}
I20260812 07:04:56.205256 20996 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d485ca153d44128aca918b986bd9de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:55 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260812 07:04:56.205981 21177 tablet_replica.cc:330] stopping tablet replica
I20260812 07:04:56.206245 21177 raft_consensus.cc:2229] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:56.206465 21177 raft_consensus.cc:2258] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:56.207417 21177 ts_tablet_manager.cc:1905] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:56.210685 21177 ts_tablet_manager.cc:1918] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:56.210769 21177 log.cc:1196] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/c7d485ca153d44128aca918b986bd9de
I20260812 07:04:56.211061 21177 ts_tablet_manager.cc:1939] T c7d485ca153d44128aca918b986bd9de P dfa773df8ace4664816635a558cff9ed: Deleting consensus metadata
I20260812 07:04:56.212685 20772 catalog_manager.cc:4841] TS dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731): tablet c7d485ca153d44128aca918b986bd9de (table test-table [id=f93311083bf649d4a30cccbbd0e0d77a]) successfully deleted
W20260812 07:04:56.291169 21166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:56.291425 21166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:56.291468 21166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:56.295408 21166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:56.295459 21166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:56.295521 21166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:04:56.300755 21166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:38777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=33813
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:56.301762 21166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:56.302738 21166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:56.312532 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:56.312609 21166 server_base.cc:1034] running on GCE node
W20260812 07:04:56.312539 21193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:56.312806 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:56.313205 21166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:56.313858 21166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:56.315165 21166 hybrid_clock.cc:648] HybridClock initialized: now 1786518296315063 us; error 47 us; skew 500 ppm
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:56.318089 21166 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:04:56.318992 21166 webserver.cc:460] Webserver started at http://127.27.110.194:33813/ using document root <none> and password file <none>
I20260812 07:04:56.319590 21166 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:56.319666 21166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:56.323577 21166 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:56.325702 21206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.327016 21166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:56.327203 21166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "c6b72a0e83ff4d5b8857601bf358d515"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "828cc4abb72281d31e5f6fc8823003ca"
server_key_iv: "30634e3d8147685a0d239fb6973ad564"
server_key_version: "encryptionkey@0"
I20260812 07:04:56.327795 21166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:56.349370 21166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:56.352473 21166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:56.352696 21166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:56.353295 21166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:56.354137 21166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:56.354185 21166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.354249 21166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:56.354288 21166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.363540 21166 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:38777
I20260812 07:04:56.363569 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:38777 every 8 connection(s)
I20260812 07:04:56.364583 21166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:56.366448 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 21166
I20260812 07:04:56.366866 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:44517
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:39337
--webserver_port=38863
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:56.379006 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.366449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:44943 (local address 127.27.110.254:39337)
0812 07:04:56.366622 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.366627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.367524 (+ 897us) server_negotiation.cc:407] Connection header received
0812 07:04:56.368926 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.368929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.368991 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.369100 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.370389 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.371017 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.372249 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.372486 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.372589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.375448 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:56.375452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:56.375453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:56.375477 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:56.376949 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.377452 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.377456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.377458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.377543 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.377884 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.377887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.377889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.378075 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:56.378196 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.378688 (+ 492us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.378827 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:04:56.379943 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.366814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:44943)
0812 07:04:56.367352 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.367407 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.368710 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.369270 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.369277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.369757 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.370237 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.370252 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.371210 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.371213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.372103 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.372113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.372654 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.372659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.372828 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.373465 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.373481 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.375276 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:56.377068 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.377074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.377089 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.377348 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.377638 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.377641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.377643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.377787 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.378196 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:56.378204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:56.378455 (+ 251us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.378645 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.378970 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":189,"threads_started":1}
I20260812 07:04:56.381142 21417 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39337
I20260812 07:04:56.381407 21417 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:56.382021 21417 heartbeater.cc:511] Master 127.27.110.254:39337 requested a full tablet report, sending...
I20260812 07:04:56.383201 20797 ts_manager.cc:194] Registered new tserver with Master: c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777)
I20260812 07:04:56.383968 20797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:44943
W20260812 07:04:56.482517 21421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:56.482765 21421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:56.482806 21421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:56.486660 21421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:56.486713 21421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:56.486768 21421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:04:56.491909 21421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=38863
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:56.492884 21421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:56.493868 21421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:56.503623 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:56.503772 21435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:56.503623 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:56.503970 21421 server_base.cc:1034] running on GCE node
I20260812 07:04:56.504357 21421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:56.504983 21421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:56.506184 21421 hybrid_clock.cc:648] HybridClock initialized: now 1786518296506173 us; error 31 us; skew 500 ppm
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:56.509194 21421 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:04:56.510074 21421 webserver.cc:460] Webserver started at http://127.27.110.195:38863/ using document root <none> and password file <none>
I20260812 07:04:56.510640 21421 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:56.510716 21421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:56.514670 21421 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:04:56.516731 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.517930 21421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:04:56.518069 21421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "fa082d3e7a21412fa5276bfefd023f19"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "4230409bd9b39f7ab0e8bc7088bc85c5"
server_key_iv: "b24755459126741614f3b24f8c2306e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:56.518627 21421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:56.545619 21421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:56.549038 21421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:56.549253 21421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:56.549871 21421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:56.550734 21421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:56.550805 21421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.550876 21421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:56.550916 21421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.560299 21421 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44517
I20260812 07:04:56.560326 21669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44517 every 8 connection(s)
I20260812 07:04:56.561300 21421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:56.564929 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 21421
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:04:56.574484 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.565687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.27.110.254:39337)
0812 07:04:56.565933 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.565937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.566070 (+ 133us) server_negotiation.cc:407] Connection header received
0812 07:04:56.566228 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.566231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.566285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.566374 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.567188 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.567855 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.569069 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.569289 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.569365 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.570980 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:56.570984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:56.570986 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:56.571009 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:56.572825 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.573249 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.573255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.573257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.573354 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.573633 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.573636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.573638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.573816 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:56.574040 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.574176 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.574307 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":77,"threads_started":1}
I20260812 07:04:56.577170 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.563144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:47103 (local address 127.27.110.254:39337)
0812 07:04:56.563314 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.563318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.564275 (+ 957us) server_negotiation.cc:407] Connection header received
0812 07:04:56.566083 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.566087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.566144 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.566234 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:56.567627 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.568210 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.569671 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.570044 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.570204 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.573105 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:56.573109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:56.573117 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:56.573138 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:56.574834 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.575561 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.575564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.575566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.575655 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:56.576015 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:56.576019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:56.576020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:56.576253 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:56.576362 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.576891 (+ 529us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.577007 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":8576}
I20260812 07:04:56.577989 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.563602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:47103)
0812 07:04:56.564111 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.564157 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.565816 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.566485 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.566495 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.566867 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:56.567438 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.567460 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.568333 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.568336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.569531 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.569543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.570244 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.570248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.570417 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.570966 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:56.570983 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:56.572935 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:56.574956 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.574962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.574976 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.575436 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.575754 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:56.575757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:56.575759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:56.575909 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:56.576364 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:56.576373 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:56.576647 (+ 274us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.576832 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.577166 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":116,"threads_started":1}
I20260812 07:04:56.579052 21672 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39337
I20260812 07:04:56.579334 21672 heartbeater.cc:464] Registering TS with master...
I20260812 07:04:56.579854 21672 heartbeater.cc:511] Master 127.27.110.254:39337 requested a full tablet report, sending...
I20260812 07:04:56.580824 20797 ts_manager.cc:194] Registered new tserver with Master: fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
I20260812 07:04:56.581377 20797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:47103
I20260812 07:04:56.587677 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:04:56.591564 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.588242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34742 (local address 127.27.110.254:39337)
0812 07:04:56.588349 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.588354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.588538 (+ 184us) server_negotiation.cc:407] Connection header received
0812 07:04:56.588666 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.588668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.588753 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.588835 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:56.589039 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.589514 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.590294 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.590502 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.590574 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.590855 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:56.591288 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:56.591375 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.591435 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.591474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 07:04:56.592129 20797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:04:56.595321 20797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:04:56.603046 20996 tablet_service.cc:1467] Processing CreateTablet for tablet 0bccad5314d54246989fcd029d36630e (DEFAULT_TABLE table=test-table [id=bd32c2c3f97a4f93aca9341363db149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:56.603747 20996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bccad5314d54246989fcd029d36630e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:56.606567 20949 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Bootstrap starting.
I20260812 07:04:56.607393 20949 tablet_bootstrap.cc:654] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:56.608304 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.601886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:38777 (local address 127.0.0.1:33482)
0812 07:04:56.601991 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.602008 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.602087 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.603017 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.603022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.603042 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:56.603177 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.603182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.604919 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.604925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.606214 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.606245 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.607777 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.607783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.607899 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.607952 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.608060 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.608148 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":22}
I20260812 07:04:56.608345 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.602084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.27.110.194:38777)
0812 07:04:56.602450 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.602467 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.602493 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:04:56.602579 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.602584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.602831 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.603348 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:56.603545 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.604795 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.606385 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.607624 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.607759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.607893 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.608085 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.608168 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":132,"threads_started":1}
I20260812 07:04:56.608994 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.602130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44517 (local address 127.0.0.1:53654)
0812 07:04:56.602432 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.602446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.602533 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.603766 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.603772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.603793 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:56.603878 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.603884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.605221 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.605224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.606351 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.606499 (+ 148us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.608568 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.608572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.608671 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.608694 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.608774 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.608854 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260812 07:04:56.609059 20949 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: No bootstrap required, opened a new log
I20260812 07:04:56.609062 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.602719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.27.110.195:44517)
0812 07:04:56.603106 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.603125 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.603150 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:04:56.603230 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.603237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.603436 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.603604 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:56.604009 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.605077 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.606620 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.607635 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.607759 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.607893 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.608823 (+ 930us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.608928 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":140,"threads_started":1}
I20260812 07:04:56.609146 20949 ts_tablet_manager.cc:1397] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:56.609586 20949 raft_consensus.cc:348] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.609674 20949 raft_consensus.cc:374] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:56.609700 20949 raft_consensus.cc:729] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa773df8ace4664816635a558cff9ed, State: Initialized, Role: FOLLOWER
I20260812 07:04:56.609833 20949 consensus_queue.cc:260] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.610097 20949 ts_tablet_manager.cc:1428] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 07:04:56.611559 21298 tablet_service.cc:1467] Processing CreateTablet for tablet 0bccad5314d54246989fcd029d36630e (DEFAULT_TABLE table=test-table [id=bd32c2c3f97a4f93aca9341363db149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:56.611850 21540 tablet_service.cc:1467] Processing CreateTablet for tablet 0bccad5314d54246989fcd029d36630e (DEFAULT_TABLE table=test-table [id=bd32c2c3f97a4f93aca9341363db149b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:04:56.612746 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bccad5314d54246989fcd029d36630e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:56.612741 21298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bccad5314d54246989fcd029d36630e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:04:56.616145 21174 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:56.616314 21174 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.617285 21174 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777), fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
I20260812 07:04:56.621857 21699 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Bootstrap starting.
I20260812 07:04:56.622061 21700 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Bootstrap starting.
I20260812 07:04:56.622749 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.617428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:38777 (local address 127.27.110.193:60063)
0812 07:04:56.617538 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.617553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.617640 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.618589 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.618592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.618609 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:56.618765 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.618776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.620396 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.620399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.621472 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.621481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.622308 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.622315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.622431 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.622481 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.622538 (+ 57us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.622622 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":19}
I20260812 07:04:56.622774 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.618244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:54705 (local address 127.27.110.195:44517)
0812 07:04:56.618358 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.618362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.618375 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:04:56.618421 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.618423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.618466 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.618546 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:56.618932 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.619923 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.621466 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.622068 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.622213 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.622284 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.622610 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.622676 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 07:04:56.623171 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.617666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44517 (local address 127.27.110.193:54705)
0812 07:04:56.617985 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.618001 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.618090 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.618646 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.618649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.618662 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:56.618809 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.618819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.620057 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.620060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.621328 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.621338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.622387 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.622392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.622470 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.622498 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.622885 (+ 387us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.622979 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":24,"thread_start_us":137,"threads_started":1}
I20260812 07:04:56.623637 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.618137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:60063 (local address 127.27.110.194:38777)
0812 07:04:56.618376 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.618381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.618394 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:04:56.618432 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.618435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.618473 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.619016 (+ 543us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:56.619596 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.620276 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.621610 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.622180 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.623193 (+ 1013us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.623244 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.623284 (+ 40us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.623364 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260812 07:04:56.624607 21699 tablet_bootstrap.cc:654] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:56.625100 21700 tablet_bootstrap.cc:654] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Neither blocks nor log segments found. Creating new log.
I20260812 07:04:56.625731 21699 log.cc:824] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:56.624007 21592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bccad5314d54246989fcd029d36630e" candidate_uuid: "dfa773df8ace4664816635a558cff9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa082d3e7a21412fa5276bfefd023f19" is_pre_election: true
I20260812 07:04:56.624727 21339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bccad5314d54246989fcd029d36630e" candidate_uuid: "dfa773df8ace4664816635a558cff9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b72a0e83ff4d5b8857601bf358d515" is_pre_election: true
I20260812 07:04:56.626082 21700 log.cc:824] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Log is configured to *not* fsync() on all Append() calls
W20260812 07:04:56.626633 20943 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): Illegal state: must be running to vote when last-logged opid is not known
W20260812 07:04:56.626861 20943 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260812 07:04:56.626942 20943 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa773df8ace4664816635a558cff9ed; no voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19
I20260812 07:04:56.627373 21174 raft_consensus.cc:2726] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 07:04:56.628047 21699 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: No bootstrap required, opened a new log
I20260812 07:04:56.628261 21699 ts_tablet_manager.cc:1397] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260812 07:04:56.628429 21700 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: No bootstrap required, opened a new log
I20260812 07:04:56.628650 21700 ts_tablet_manager.cc:1397] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 07:04:56.630465 21699 raft_consensus.cc:348] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.630703 21699 raft_consensus.cc:374] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:56.630769 21699 raft_consensus.cc:729] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa082d3e7a21412fa5276bfefd023f19, State: Initialized, Role: FOLLOWER
I20260812 07:04:56.630684 21700 raft_consensus.cc:348] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.630947 21700 raft_consensus.cc:374] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:04:56.631027 21700 raft_consensus.cc:729] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b72a0e83ff4d5b8857601bf358d515, State: Initialized, Role: FOLLOWER
I20260812 07:04:56.631263 21699 consensus_queue.cc:260] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.631544 21700 consensus_queue.cc:260] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.631945 21672 heartbeater.cc:503] Master 127.27.110.254:39337 was elected leader, sending a full tablet report...
I20260812 07:04:56.632226 21417 heartbeater.cc:503] Master 127.27.110.254:39337 was elected leader, sending a full tablet report...
I20260812 07:04:56.632376 21699 ts_tablet_manager.cc:1428] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260812 07:04:56.632513 21700 ts_tablet_manager.cc:1428] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260812 07:04:56.664819 21158 tablet.cc:2369] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:04:56.675618 21706 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:56.675835 21706 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.676908 21706 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517), dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731)
I20260812 07:04:56.681696 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.677253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44517 (local address 127.27.110.194:33163)
0812 07:04:56.677416 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.677435 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.677579 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.677958 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.677963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.677987 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:56.678149 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.678159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.679203 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.679205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.680435 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.680447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.681280 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.681282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.681365 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.681386 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.681446 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.681549 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":2}
I20260812 07:04:56.681705 21696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.677345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:33163 (local address 127.27.110.195:44517)
0812 07:04:56.677527 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.677533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.677550 (+ 17us) server_negotiation.cc:407] Connection header received
0812 07:04:56.677711 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.677715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.677765 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.677852 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:56.678265 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.679103 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.680573 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.681162 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.681288 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.681343 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.681492 (+ 149us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.681567 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:04:56.681999 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.677766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:46595 (local address 127.27.110.193:37731)
0812 07:04:56.677921 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.677926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.678101 (+ 175us) server_negotiation.cc:407] Connection header received
0812 07:04:56.678223 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.678227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.678277 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.678356 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:04:56.678619 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.679354 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.680735 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.681418 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.681552 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.681683 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.681807 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.681893 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:04:56.682013 21709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.677592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:37731 (local address 127.27.110.194:46595)
0812 07:04:56.678006 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.678020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:56.678125 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:56.678376 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:56.678378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:56.678394 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:04:56.678513 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.678517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.679470 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.679474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.680625 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:56.680633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.681574 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.681577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:56.681676 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.681698 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:04:56.681771 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 07:04:56.681846 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":26,"thread_start_us":183,"threads_started":1}
I20260812 07:04:56.682394 21592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bccad5314d54246989fcd029d36630e" candidate_uuid: "c6b72a0e83ff4d5b8857601bf358d515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa082d3e7a21412fa5276bfefd023f19" is_pre_election: true
I20260812 07:04:56.682498 21065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bccad5314d54246989fcd029d36630e" candidate_uuid: "c6b72a0e83ff4d5b8857601bf358d515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa773df8ace4664816635a558cff9ed" is_pre_election: true
I20260812 07:04:56.682677 21592 raft_consensus.cc:2454] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b72a0e83ff4d5b8857601bf358d515 in term 0.
I20260812 07:04:56.682807 21065 raft_consensus.cc:2454] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b72a0e83ff4d5b8857601bf358d515 in term 0.
I20260812 07:04:56.683120 21208 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19; no voters:
I20260812 07:04:56.683377 21706 raft_consensus.cc:2781] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:04:56.683461 21706 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:04:56.683499 21706 raft_consensus.cc:3037] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:56.684608 21706 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.684947 21706 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [CANDIDATE]: Term 1 election: Requested vote from peers fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517), dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731)
I20260812 07:04:56.685343 21065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bccad5314d54246989fcd029d36630e" candidate_uuid: "c6b72a0e83ff4d5b8857601bf358d515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfa773df8ace4664816635a558cff9ed"
I20260812 07:04:56.685336 21592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bccad5314d54246989fcd029d36630e" candidate_uuid: "c6b72a0e83ff4d5b8857601bf358d515" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa082d3e7a21412fa5276bfefd023f19"
I20260812 07:04:56.685467 21065 raft_consensus.cc:3037] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:56.685467 21592 raft_consensus.cc:3037] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:04:56.686772 21065 raft_consensus.cc:2454] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b72a0e83ff4d5b8857601bf358d515 in term 1.
I20260812 07:04:56.686772 21592 raft_consensus.cc:2454] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b72a0e83ff4d5b8857601bf358d515 in term 1.
I20260812 07:04:56.687040 21208 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19; no voters:
I20260812 07:04:56.687258 21706 raft_consensus.cc:2781] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:04:56.687502 21706 raft_consensus.cc:686] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 1 LEADER]: Becoming Leader. State: Replica: c6b72a0e83ff4d5b8857601bf358d515, State: Running, Role: LEADER
I20260812 07:04:56.687853 21706 consensus_queue.cc:237] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:56.691176 20797 catalog_manager.cc:5495] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194). New cstate: current_term: 1 leader_uuid: "c6b72a0e83ff4d5b8857601bf358d515" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } health_report { overall_health: UNKNOWN } } }
I20260812 07:04:56.716946 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.713294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.27.110.194:38777)
0812 07:04:56.713440 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:56.713445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:56.713461 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:04:56.713619 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:56.713622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:56.713678 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:56.713787 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:04:56.714023 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.714524 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.715505 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:56.715722 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:56.715826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:56.716135 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:04:56.716553 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:04:56.716689 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:56.716733 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 07:04:56.716785 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 07:04:56.722918 21592 raft_consensus.cc:1261] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Refusing update from remote peer c6b72a0e83ff4d5b8857601bf358d515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:56.723090 21065 raft_consensus.cc:1261] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Refusing update from remote peer c6b72a0e83ff4d5b8857601bf358d515: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:04:56.723711 21710 consensus_queue.cc:1035] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:56.723888 21706 consensus_queue.cc:1035] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:04:56.738097 20795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:56.738283 20795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:04:56.740752 20795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Sending DeleteTablet for 3 replicas of tablet 0bccad5314d54246989fcd029d36630e
I20260812 07:04:56.741442 21298 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bccad5314d54246989fcd029d36630e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:56 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:33482
I20260812 07:04:56.741611 21540 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bccad5314d54246989fcd029d36630e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:56 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:53654
I20260812 07:04:56.741919 20996 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bccad5314d54246989fcd029d36630e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:04:56 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:47434
I20260812 07:04:56.742105 21738 tablet_replica.cc:330] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: stopping tablet replica
I20260812 07:04:56.742303 21739 tablet_replica.cc:330] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: stopping tablet replica
I20260812 07:04:56.742486 21739 raft_consensus.cc:2229] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:56.742499 21738 raft_consensus.cc:2229] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:04:56.742563 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 20875
I20260812 07:04:56.742717 21739 raft_consensus.cc:2258] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:56.758625 21737 tablet_replica.cc:330] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: stopping tablet replica
I20260812 07:04:56.759169 21737 raft_consensus.cc:2229] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:04:56.759660 21737 raft_consensus.cc:2258] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:56.765358 21737 ts_tablet_manager.cc:1905] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:04:56.768424 21737 ts_tablet_manager.cc:1918] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:56.768553 21737 log.cc:1196] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/0bccad5314d54246989fcd029d36630e
I20260812 07:04:56.768932 21737 ts_tablet_manager.cc:1939] T 0bccad5314d54246989fcd029d36630e P c6b72a0e83ff4d5b8857601bf358d515: Deleting consensus metadata
I20260812 07:04:56.770430 21738 raft_consensus.cc:2258] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:04:56.770375 20772 catalog_manager.cc:4841] TS c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): tablet 0bccad5314d54246989fcd029d36630e (table test-table [id=bd32c2c3f97a4f93aca9341363db149b]) successfully deleted
I20260812 07:04:56.772190 21738 ts_tablet_manager.cc:1905] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 07:04:56.773144 20772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:37731 (error 108)
I20260812 07:04:56.773499 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 21166
I20260812 07:04:56.776273 21738 ts_tablet_manager.cc:1918] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:04:56.776463 21738 log.cc:1196] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/0bccad5314d54246989fcd029d36630e
I20260812 07:04:56.776885 21738 ts_tablet_manager.cc:1939] T 0bccad5314d54246989fcd029d36630e P fa082d3e7a21412fa5276bfefd023f19: Deleting consensus metadata
I20260812 07:04:56.777318 21694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:56.776668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:37731 (local address 127.0.0.1:47444)
0812 07:04:56.777178 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:56.777241 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:37731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429}
W20260812 07:04:56.780162 20772 catalog_manager.cc:4584] TS dfa773df8ace4664816635a558cff9ed (127.27.110.193:37731): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bccad5314d54246989fcd029d36630e: Network error: Client connection negotiation failed: client connection to 127.27.110.193:37731: connect: Connection refused (error 111)
I20260812 07:04:56.781096 20772 catalog_manager.cc:4841] TS fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): tablet 0bccad5314d54246989fcd029d36630e (table test-table [id=bd32c2c3f97a4f93aca9341363db149b]) successfully deleted
I20260812 07:04:56.785828 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 21421
I20260812 07:04:56.799908 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 20744
I20260812 07:04:56.817054 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39337
--webserver_interface=127.27.110.254
--webserver_port=45647
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:56.932862 21749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:56.933120 21749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:56.933158 21749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:56.937176 21749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:04:56.937232 21749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:56.937252 21749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:56.937268 21749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:04:56.937284 21749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:04:56.942396 21749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:56.943411 21749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:56.944414 21749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:56.953588 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:56.953718 21749 server_base.cc:1034] running on GCE node
W20260812 07:04:56.953604 21756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:56.953583 21762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:56.954473 21749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:56.955598 21749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:56.956785 21749 hybrid_clock.cc:648] HybridClock initialized: now 1786518296956770 us; error 27 us; skew 500 ppm
Aug 12 07:04:56 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518296, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:56.959605 21749 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:04:56.960475 21749 webserver.cc:460] Webserver started at http://127.27.110.254:45647/ using document root <none> and password file <none>
I20260812 07:04:56.960976 21749 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:56.961041 21749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:56.964900 21749 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:56.967041 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:56.968351 21749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:56.968451 21749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "781082ca25c8457a8db49f550f28a84a"
format_stamp: "Formatted at 2026-08-12 07:04:54 on dist-test-slave-w4v5"
server_key: "07a75292c728b68c554bcfbd7a4fa74a"
server_key_iv: "12018efdd2d8c27b5f043be775200835"
server_key_version: "encryptionkey@0"
I20260812 07:04:56.968963 21749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:56.980662 21749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:56.983700 21749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:56.983906 21749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:56.991374 21749 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39337
I20260812 07:04:56.991677 21873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39337 every 8 connection(s)
I20260812 07:04:56.992578 21749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:04:56.994477 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 21749
I20260812 07:04:56.994961 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:37731
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:39337
--webserver_port=38617
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:56.997848 21876 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:04:56.999591 21876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap starting.
I20260812 07:04:57.017849 21876 log.cc:824] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:57.029078 21876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:57.029495 21876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a: Bootstrap complete.
I20260812 07:04:57.033731 21876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:57.034109 21876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781082ca25c8457a8db49f550f28a84a, State: Initialized, Role: FOLLOWER
I20260812 07:04:57.034595 21876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:57.034709 21876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:04:57.034793 21876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:04:57.034884 21876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 2 FOLLOWER]: Advancing to term 3
I20260812 07:04:57.036828 21876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:57.037158 21876 leader_election.cc:302] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 781082ca25c8457a8db49f550f28a84a; no voters:
I20260812 07:04:57.037520 21876 leader_election.cc:288] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 07:04:57.037597 21883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 3 FOLLOWER]: Leader election won for term 3
I20260812 07:04:57.037832 21883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [term 3 LEADER]: Becoming Leader. State: Replica: 781082ca25c8457a8db49f550f28a84a, State: Running, Role: LEADER
I20260812 07:04:57.038185 21883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } }
I20260812 07:04:57.038769 21876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:04:57.039217 21887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 781082ca25c8457a8db49f550f28a84a. Latest consensus state: current_term: 3 leader_uuid: "781082ca25c8457a8db49f550f28a84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } } }
I20260812 07:04:57.039342 21887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:57.039594 21885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "781082ca25c8457a8db49f550f28a84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "781082ca25c8457a8db49f550f28a84a" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39337 } } }
I20260812 07:04:57.039717 21885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 781082ca25c8457a8db49f550f28a84a [sys.catalog]: This master's current role is: LEADER
I20260812 07:04:57.040371 21890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:04:57.045595 21890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:04:57.046327 21890 catalog_manager.cc:1259] Loaded cluster ID: 30c5e06cf18842a59b7b96af20826814
I20260812 07:04:57.046408 21890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 07:04:57.047389 21890 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc421fa29ad google::LogMessage::Fail() at ??:0
@ 0x7fc421fa588a google::LogMessage::SendToLog() at ??:0
@ 0x7fc421fa24de google::LogMessage::Flush() at ??:0
@ 0x7fc421fa3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc42c8e1fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fc42c8e2d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fc42c8df579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fc42c923640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc4241f43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fc4241f4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fc4241f63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc4241e5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc4258296db start_thread at ??:0
@ 0x7fc420aa061f clone at ??:0
W20260812 07:04:57.140213 21879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:57.140448 21879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:57.140491 21879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:57.144534 21879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:57.144590 21879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:57.144673 21879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:04:57.149825 21879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:37731
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=38617
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:57.150861 21879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:57.151899 21879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:57.161440 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:57.161422 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:57.161585 21879 server_base.cc:1034] running on GCE node
W20260812 07:04:57.161412 21958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:57.162110 21879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:57.162774 21879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:57.163955 21879 hybrid_clock.cc:648] HybridClock initialized: now 1786518297163927 us; error 45 us; skew 500 ppm
Aug 12 07:04:57 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:57.166772 21879 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:04:57.167692 21879 webserver.cc:460] Webserver started at http://127.27.110.193:38617/ using document root <none> and password file <none>
I20260812 07:04:57.168236 21879 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:57.168298 21879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:57.172417 21879 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:04:57.174647 21976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.176072 21879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:04:57.176194 21879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "dfa773df8ace4664816635a558cff9ed"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "218b6494137171ad7c32b80fb620b6cc"
server_key_iv: "25aa5e2d07c7c505632b3fc06fe31dd3"
server_key_version: "encryptionkey@0"
I20260812 07:04:57.176891 21879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:57.189553 21879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:57.192688 21879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:57.192884 21879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:57.193442 21879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:57.194494 21993 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 07:04:57.199688 21879 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 07:04:57.199759 21879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260812 07:04:57.199836 21879 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 07:04:57.201429 21879 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 07:04:57.201495 21879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260812 07:04:57.201792 21993 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Bootstrap starting.
I20260812 07:04:57.212323 21879 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:37731
I20260812 07:04:57.212567 22203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:37731 every 8 connection(s)
I20260812 07:04:57.213434 21879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:04:57.223439 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 21879
I20260812 07:04:57.223835 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:38777
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:39337
--webserver_port=33813
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:04:57.224298 21993 log.cc:824] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Log is configured to *not* fsync() on all Append() calls
I20260812 07:04:57.234227 21993 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:04:57.234710 21993 tablet_bootstrap.cc:492] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Bootstrap complete.
I20260812 07:04:57.236471 21993 ts_tablet_manager.cc:1397] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.007s
I20260812 07:04:57.240146 21993 raft_consensus.cc:348] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:57.240689 21993 raft_consensus.cc:729] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfa773df8ace4664816635a558cff9ed, State: Initialized, Role: FOLLOWER
I20260812 07:04:57.241211 21993 consensus_queue.cc:260] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:57.242245 21993 ts_tablet_manager.cc:1428] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260812 07:04:57.281361 22208 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:04:57.215405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:48147)
0812 07:04:57.215872 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.215924 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:57.216855 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:57.280447 (+ 63592us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":141,"threads_started":1}
W20260812 07:04:57.281906 22204 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:39337 (0 consecutive failures): Network error: Failed to ping master at 127.27.110.254:39337: Client connection negotiation failed: client connection to 127.27.110.254:39337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 07:04:57.282699 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.282439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:34629)
0812 07:04:57.282571 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.282644 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 07:04:57.283366 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.283185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:33823)
0812 07:04:57.283279 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.283321 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 07:04:57.283536 22204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:04:57.345825 22211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:57.346096 22211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:57.346139 22211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:57.350210 22211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:57.350276 22211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:57.350335 22211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:04:57.355731 22211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:38777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=33813
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:57.356740 22211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:57.357910 22211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:57.367575 22221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:57.367677 22225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:57.367592 22223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:57.367659 22211 server_base.cc:1034] running on GCE node
I20260812 07:04:57.368219 22211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:57.368851 22211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:57.370013 22211 hybrid_clock.cc:648] HybridClock initialized: now 1786518297369990 us; error 33 us; skew 500 ppm
Aug 12 07:04:57 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:57.372895 22211 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:04:57.373809 22211 webserver.cc:460] Webserver started at http://127.27.110.194:33813/ using document root <none> and password file <none>
I20260812 07:04:57.374362 22211 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:57.374425 22211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:57.378249 22211 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:57.380462 22240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.381914 22211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 07:04:57.382059 22211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "c6b72a0e83ff4d5b8857601bf358d515"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "828cc4abb72281d31e5f6fc8823003ca"
server_key_iv: "30634e3d8147685a0d239fb6973ad564"
server_key_version: "encryptionkey@0"
I20260812 07:04:57.382620 22211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:57.397957 22211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:57.400957 22211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:57.401167 22211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:57.401777 22211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:57.402626 22211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:57.402696 22211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.402765 22211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:57.402812 22211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.412050 22211 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:38777
I20260812 07:04:57.412087 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:38777 every 8 connection(s)
I20260812 07:04:57.413028 22211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:04:57.416680 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.415039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:47469)
0812 07:04:57.415553 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.415992 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":110,"threads_started":1}
W20260812 07:04:57.417155 22442 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:39337 (0 consecutive failures): Network error: Failed to ping master at 127.27.110.254:39337: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
I20260812 07:04:57.417698 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.417475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:33103)
0812 07:04:57.417580 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.417637 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260812 07:04:57.418279 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.418079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:34223)
0812 07:04:57.418174 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.418222 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260812 07:04:57.418444 22442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 07:04:57.423048 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 22211
I20260812 07:04:57.423509 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:44517
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:39337
--webserver_port=38863
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:04:57.535142 22452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:04:57.535380 22452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:04:57.535423 22452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:04:57.539333 22452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:04:57.539382 22452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:04:57.539438 22452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:04:57.544425 22452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=38863
--tserver_master_addrs=127.27.110.254:39337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:04:57.545331 22452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:04:57.546243 22452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:04:57.555578 22458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:57.555545 22461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:04:57.555645 22459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:04:57.555879 22452 server_base.cc:1034] running on GCE node
I20260812 07:04:57.556253 22452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:04:57.556878 22452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:04:57.558055 22452 hybrid_clock.cc:648] HybridClock initialized: now 1786518297558036 us; error 26 us; skew 500 ppm
Aug 12 07:04:57 dist-test-slave-w4v5 krb5kdc[19782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:04:57.560909 22452 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:04:57.561793 22452 webserver.cc:460] Webserver started at http://127.27.110.195:38863/ using document root <none> and password file <none>
I20260812 07:04:57.562348 22452 fs_manager.cc:359] Metadata directory not provided
I20260812 07:04:57.562412 22452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:04:57.566267 22452 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:04:57.568394 22477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.569676 22452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:04:57.569823 22452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "fa082d3e7a21412fa5276bfefd023f19"
format_stamp: "Formatted at 2026-08-12 07:04:55 on dist-test-slave-w4v5"
server_key: "4230409bd9b39f7ab0e8bc7088bc85c5"
server_key_iv: "b24755459126741614f3b24f8c2306e7"
server_key_version: "encryptionkey@0"
I20260812 07:04:57.570389 22452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:04:57.590775 22452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:04:57.593515 22452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:04:57.593713 22452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:04:57.594297 22452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:04:57.595144 22452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:04:57.595211 22452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.595283 22452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:04:57.595324 22452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:04:57.604600 22452 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44517
I20260812 07:04:57.604642 22698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44517 every 8 connection(s)
I20260812 07:04:57.605598 22452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:04:57.609236 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.607660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:55697)
0812 07:04:57.608130 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.608548 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
W20260812 07:04:57.609658 22700 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:39337 (0 consecutive failures): Network error: Failed to ping master at 127.27.110.254:39337: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
I20260812 07:04:57.610240 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.609984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:44647)
0812 07:04:57.610119 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.610173 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260812 07:04:57.610716 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 22452
I20260812 07:04:57.610939 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:57.610714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:39757)
0812 07:04:57.610847 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:57.610896 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260812 07:04:57.611191 22700 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 07:04:58.284571 22705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.284138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:50627)
0812 07:04:58.284395 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.284454 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
I20260812 07:04:58.419454 22707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.418957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:51099)
0812 07:04:58.419261 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.419320 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260812 07:04:58.612150 22709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.611775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:41143)
0812 07:04:58.612003 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.612055 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260812 07:04:58.911201 22712 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:04:58.911438 22712 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:04:58.912519 22712 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777), fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
Aug 12 07:04:58 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.195@KRBTEST.COM
Aug 12 07:04:58 dist-test-slave-w4v5 krb5kdc[19782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518297, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.194@KRBTEST.COM
I20260812 07:04:58.926836 22715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.912802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:38777 (local address 127.27.110.193:33329)
0812 07:04:58.913040 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.913055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:58.913160 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:58.914939 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:58.914945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:58.915574 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:58.916068 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.916088 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.917391 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.917394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.918242 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.918253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.919055 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.919059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.919189 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.919858 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:58.919876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:58.922110 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:58.924448 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:58.924451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:58.924453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:58.924732 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:58.925157 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:58.925161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:58.925163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:58.925295 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:58.925900 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:58.925902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:58.926079 (+ 177us) client_negotiation.cc:769] Sending connection context
0812 07:04:58.926267 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 07:04:58.926627 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":99,"spinlock_wait_cycles":12544,"thread_start_us":87,"threads_started":1}
I20260812 07:04:58.926859 22716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.912919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:33329 (local address 127.27.110.194:38777)
0812 07:04:58.913353 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:58.913382 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:58.913431 (+ 49us) server_negotiation.cc:407] Connection header received
0812 07:04:58.914194 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:58.914200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:58.914553 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:58.914853 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:58.916271 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.917249 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.918396 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.918905 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.919142 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.922344 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:58.922352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:58.922368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:58.922408 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:58.924310 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:58.924850 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:58.924855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:58.924860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:58.925007 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:58.925392 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:58.925395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:58.925396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:58.925750 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:58.925903 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:58.926378 (+ 475us) server_negotiation.cc:299] Negotiation successful
0812 07:04:58.926673 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":87,"threads_started":1}
I20260812 07:04:58.927291 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.913187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44517 (local address 127.27.110.193:48891)
0812 07:04:58.913520 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:58.913536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:04:58.913638 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:04:58.914955 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:04:58.914959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:04:58.915561 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:04:58.916070 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.916087 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.917405 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.917410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.918236 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:04:58.918245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.918938 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.918942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:04:58.919144 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.919900 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:04:58.919913 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:04:58.921561 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:04:58.924067 (+ 2506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:58.924075 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:58.924090 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:58.924428 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:58.924835 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:04:58.924838 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:04:58.924840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:04:58.925007 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:04:58.925831 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:04:58.925841 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:04:58.926079 (+ 238us) client_negotiation.cc:769] Sending connection context
0812 07:04:58.926267 (+ 188us) client_negotiation.cc:240] Negotiation successful
0812 07:04:58.927092 (+ 825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":579,"spinlock_wait_cycles":3840,"thread_start_us":57,"threads_started":1}
I20260812 07:04:58.927817 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:58.913377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:48891 (local address 127.27.110.195:44517)
0812 07:04:58.913694 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 07:04:58.913718 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 07:04:58.913747 (+ 29us) server_negotiation.cc:407] Connection header received
0812 07:04:58.914370 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:04:58.914376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:04:58.914679 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:04:58.915009 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:04:58.916325 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.917277 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.918364 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:04:58.918816 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:04:58.919001 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:04:58.921814 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:04:58.921822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:04:58.921839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:04:58.921875 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:04:58.923927 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:58.924562 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:58.924571 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:58.924577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:58.924726 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:04:58.925157 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:04:58.925161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:04:58.925163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:04:58.925703 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:04:58.925819 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 07:04:58.926663 (+ 844us) server_negotiation.cc:299] Negotiation successful
0812 07:04:58.927678 (+ 1015us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":125,"threads_started":1}
W20260812 07:04:58.929714 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
W20260812 07:04:58.929970 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
I20260812 07:04:58.930032 21977 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa773df8ace4664816635a558cff9ed; no voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19
I20260812 07:04:58.930295 22712 raft_consensus.cc:2726] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:04:59.285765 22717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:59.285331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:34883)
0812 07:04:59.285596 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:59.285655 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260812 07:04:59.420557 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:59.420117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:57829)
0812 07:04:59.420398 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:59.420454 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
I20260812 07:04:59.613233 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:04:59.612835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:45973)
0812 07:04:59.613098 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 07:04:59.613148 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":102,"threads_started":1}
I20260812 07:05:00.287019 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:00.286564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:35417)
0812 07:05:00.286850 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:00.286912 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260812 07:05:00.421706 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:00.421298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:33873)
0812 07:05:00.421544 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:00.421601 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260812 07:05:00.607784 22723 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:00.607896 22723 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:05:00.608431 22723 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777), fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
W20260812 07:05:00.609297 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
W20260812 07:05:00.609468 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
I20260812 07:05:00.609547 21977 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa773df8ace4664816635a558cff9ed; no voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19
I20260812 07:05:00.609714 22723 raft_consensus.cc:2726] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:05:00.614398 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:00.613931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:56143)
0812 07:05:00.614179 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:00.614255 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260812 07:05:01.288335 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:01.287848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:35435)
0812 07:05:01.288150 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:01.288229 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":111,"threads_started":1}
I20260812 07:05:01.422888 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:01.422426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:60257)
0812 07:05:01.422734 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:01.422786 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":114,"threads_started":1}
I20260812 07:05:01.615726 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:01.615268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:55901)
0812 07:05:01.615566 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:01.615630 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260812 07:05:02.289816 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:02.289345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:46743)
0812 07:05:02.289648 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:02.289713 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":78,"threads_started":1}
I20260812 07:05:02.424129 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:02.423705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:39815)
0812 07:05:02.423963 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:02.424022 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260812 07:05:02.617004 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:02.616543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:60299)
0812 07:05:02.616840 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:02.616900 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260812 07:05:03.291118 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.290660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:46857)
0812 07:05:03.290935 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:03.291001 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260812 07:05:03.425225 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.424830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:48515)
0812 07:05:03.425072 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:03.425129 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":80,"threads_started":1}
I20260812 07:05:03.592408 22740 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:03.592557 22740 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:05:03.593040 22740 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777), fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
W20260812 07:05:03.593854 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
W20260812 07:05:03.593986 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
I20260812 07:05:03.594026 21977 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa773df8ace4664816635a558cff9ed; no voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19
I20260812 07:05:03.594163 22740 raft_consensus.cc:2726] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:05:03.618255 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:03.617828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:60113)
0812 07:05:03.618099 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:03.618156 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260812 07:05:04.292586 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.292067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:42381)
0812 07:05:04.292390 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:04.292455 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":116,"threads_started":1}
I20260812 07:05:04.426447 22743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.425956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:54267)
0812 07:05:04.426300 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:04.426357 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":93,"threads_started":1}
I20260812 07:05:04.619452 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:04.618955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:34949)
0812 07:05:04.619305 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:04.619362 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260812 07:05:05.293840 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.293380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:36229)
0812 07:05:05.293684 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:05.293741 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
I20260812 07:05:05.427759 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.427304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:33483)
0812 07:05:05.427593 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:05.427658 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260812 07:05:05.493481 22750 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:05.493626 22750 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:05:05.494097 22750 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777), fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
W20260812 07:05:05.494990 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
W20260812 07:05:05.495216 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
I20260812 07:05:05.495283 21977 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa773df8ace4664816635a558cff9ed; no voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19
I20260812 07:05:05.495478 22750 raft_consensus.cc:2726] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:05:05.620615 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:05.620168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:47441)
0812 07:05:05.620453 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:05.620517 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":86,"threads_started":1}
I20260812 07:05:06.295128 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:06.294685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:57991)
0812 07:05:06.294943 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:06.295003 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":87,"threads_started":1}
I20260812 07:05:06.428979 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:06.428580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:33545)
0812 07:05:06.428834 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:06.428889 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260812 07:05:06.621803 22759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:06.621353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:42553)
0812 07:05:06.621647 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:06.621705 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":102,"threads_started":1}
I20260812 07:05:07.296590 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.296124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:57481)
0812 07:05:07.296408 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:07.296483 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260812 07:05:07.430188 22761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.429763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:33921)
0812 07:05:07.430032 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:07.430091 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260812 07:05:07.623014 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:07.622558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:45125)
0812 07:05:07.622839 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:07.622899 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260812 07:05:08.297906 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.297458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:41347)
0812 07:05:08.297745 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:08.297807 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260812 07:05:08.431341 22767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.430865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:57859)
0812 07:05:08.431187 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:08.431244 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":72,"threads_started":1}
I20260812 07:05:08.624404 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:08.623905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:33467)
0812 07:05:08.624238 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:08.624302 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260812 07:05:09.299271 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:09.298783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:39905)
0812 07:05:09.299062 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:09.299171 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260812 07:05:09.432492 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:09.432067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:37805)
0812 07:05:09.432323 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:09.432380 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260812 07:05:09.625816 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:09.625315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:35191)
0812 07:05:09.625645 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:09.625720 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":137,"threads_started":1}
I20260812 07:05:10.300683 22779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:10.300189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:43927)
0812 07:05:10.300506 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:10.300576 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":94,"threads_started":1}
I20260812 07:05:10.433691 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:10.433264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:48191)
0812 07:05:10.433530 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:10.433591 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260812 07:05:10.627144 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:10.626641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:58987)
0812 07:05:10.626917 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:10.626979 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260812 07:05:11.302069 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.301625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:46655)
0812 07:05:11.301911 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:11.301965 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":100,"threads_started":1}
I20260812 07:05:11.434885 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.434445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:44145)
0812 07:05:11.434725 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:11.434786 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":81,"threads_started":1}
I20260812 07:05:11.503345 22787 raft_consensus.cc:482] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:11.503515 22787 raft_consensus.cc:504] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6b72a0e83ff4d5b8857601bf358d515" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 38777 } } peers { permanent_uuid: "fa082d3e7a21412fa5276bfefd023f19" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44517 } } peers { permanent_uuid: "dfa773df8ace4664816635a558cff9ed" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 37731 } }
I20260812 07:05:11.503926 22787 leader_election.cc:288] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777), fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517)
W20260812 07:05:11.504848 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6b72a0e83ff4d5b8857601bf358d515 (127.27.110.194:38777): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
W20260812 07:05:11.504985 21977 leader_election.cc:341] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fa082d3e7a21412fa5276bfefd023f19 (127.27.110.195:44517): Not found: Tablet not found: 0bccad5314d54246989fcd029d36630e
I20260812 07:05:11.505046 21977 leader_election.cc:302] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dfa773df8ace4664816635a558cff9ed; no voters: c6b72a0e83ff4d5b8857601bf358d515, fa082d3e7a21412fa5276bfefd023f19
I20260812 07:05:11.505239 22787 raft_consensus.cc:2726] T 0bccad5314d54246989fcd029d36630e P dfa773df8ace4664816635a558cff9ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:05:11.628595 22789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:11.628116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:45079)
0812 07:05:11.628418 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:11.628473 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
I20260812 07:05:12.303745 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.303278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.193:59915)
0812 07:05:12.303569 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.303634 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260812 07:05:12.436192 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.435745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.194:57427)
0812 07:05:12.436035 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.436094 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260812 07:05:12.611595 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 21879
I20260812 07:05:12.627542 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 22211
I20260812 07:05:12.632157 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:12.631566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39337 (local address 127.27.110.195:47435)
0812 07:05:12.631931 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:12.632011 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:39337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":79,"threads_started":1}
I20260812 07:05:12.637010 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 22452
2026-08-12T07:05:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19463 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:05:12 dist-test-slave-w4v5 krb5kdc[22798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:05:12 dist-test-slave-w4v5 krb5kdc[22798](info): set up 2 sockets
Aug 12 07:05:12 dist-test-slave-w4v5 krb5kdc[22798](info): commencing operation
krb5kdc: starting...
W20260812 07:05:13.951447 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:05:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:05:14.087062 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42957
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:14.199481 22821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:14.199791 22821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:14.199864 22821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:14.203879 22821 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 07:05:14.203950 22821 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 07:05:14.203977 22821 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 07:05:14.204006 22821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:05:14.204063 22821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:14.204124 22821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:14.204167 22821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:05:14.204202 22821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:05:14.209415 22821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:14.210490 22821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:14.211582 22821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:14.220395 22830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:14.220410 22833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:14.220465 22821 server_base.cc:1034] running on GCE node
W20260812 07:05:14.220410 22829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:14.221194 22821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:14.222355 22821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:14.223546 22821 hybrid_clock.cc:648] HybridClock initialized: now 1786518314223514 us; error 44 us; skew 500 ppm
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:14.226439 22821 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:05:14.227451 22821 webserver.cc:460] Webserver started at http://127.27.110.254:36065/ using document root <none> and password file <none>
I20260812 07:05:14.228087 22821 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:14.228189 22821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:14.228437 22821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:14.230580 22821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "572fb8ecc78a49e2b08fd6c24f9439d1"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "ef9e2f0be3df293c8815396cf4fe0043"
server_key_iv: "3afd6c2b1c963bae77d4c5674a6ff9ce"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.231276 22821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "572fb8ecc78a49e2b08fd6c24f9439d1"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "ef9e2f0be3df293c8815396cf4fe0043"
server_key_iv: "3afd6c2b1c963bae77d4c5674a6ff9ce"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.235383 22821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:05:14.238163 22844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.240039 22821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:14.240207 22821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "572fb8ecc78a49e2b08fd6c24f9439d1"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "ef9e2f0be3df293c8815396cf4fe0043"
server_key_iv: "3afd6c2b1c963bae77d4c5674a6ff9ce"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.240348 22821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:14.256242 22821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:14.259745 22821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:14.260020 22821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:14.267424 22821 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42957
I20260812 07:05:14.267740 22953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42957 every 8 connection(s)
I20260812 07:05:14.268527 22821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:05:14.271474 22954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:14.274389 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 22821
I20260812 07:05:14.274618 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:05:14.274962 28091 external_mini_cluster.cc:1428] Setting key c5b40521c9f50316a23f1346ded42a69
I20260812 07:05:14.277724 22954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap starting.
I20260812 07:05:14.280483 22954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:14.281476 22954 log.cc:824] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:14.283725 22954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: No bootstrap required, opened a new log
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:14.288267 22954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:14.288595 22954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:14.288661 22954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572fb8ecc78a49e2b08fd6c24f9439d1, State: Initialized, Role: FOLLOWER
I20260812 07:05:14.289203 22954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:14.289395 22954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:05:14.289479 22954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:05:14.289583 22954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:14.290891 22954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:14.291311 22954 leader_election.cc:302] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 572fb8ecc78a49e2b08fd6c24f9439d1; no voters:
I20260812 07:05:14.291708 22954 leader_election.cc:288] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:05:14.291826 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:14.276462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.27.110.254:42957)
0812 07:05:14.276918 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:14.276936 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:14.276975 (+ 39us) server_negotiation.cc:407] Connection header received
0812 07:05:14.277644 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:14.277664 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:14.278066 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:14.278370 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:14.279376 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.280424 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.281732 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.282217 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.282409 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:14.285341 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:14.285348 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:14.285365 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:14.285399 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:14.288239 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:14.288757 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:14.288765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:14.288771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:14.288915 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:14.289305 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:14.289309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:14.289311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:14.289890 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:14.290113 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:14.290382 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 07:05:14.290725 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":158,"threads_started":1}
I20260812 07:05:14.291924 22962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:14.292177 22962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 LEADER]: Becoming Leader. State: Replica: 572fb8ecc78a49e2b08fd6c24f9439d1, State: Running, Role: LEADER
I20260812 07:05:14.292591 22962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:14.293043 22954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:05:14.293910 22962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 572fb8ecc78a49e2b08fd6c24f9439d1. Latest consensus state: current_term: 1 leader_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } } }
I20260812 07:05:14.294742 22962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:14.294626 22964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } } }
I20260812 07:05:14.295104 22964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:14.295255 22985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:05:14.298362 22985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:05:14.304597 22985 catalog_manager.cc:1347] Generated new cluster ID: 4f95bdc1595f4b8b81a2271d12462696
I20260812 07:05:14.304726 22985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:05:14.326664 22985 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:05:14.328197 22985 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:05:14.343989 22985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Generated new TSK 0
I20260812 07:05:14.344889 22985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:05:14.378259 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42957
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:14.494493 22997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:14.494776 22997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:14.494866 22997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:14.498870 22997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:14.498925 22997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:14.499009 22997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:05:14.504209 22997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:14.505336 22997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:14.506367 22997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:14.515581 23005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:14.515596 23006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:14.515959 23010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:14.516201 22997 server_base.cc:1034] running on GCE node
I20260812 07:05:14.516593 22997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:14.517216 22997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:14.518404 22997 hybrid_clock.cc:648] HybridClock initialized: now 1786518314518390 us; error 24 us; skew 500 ppm
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:14.521337 22997 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:05:14.522276 22997 webserver.cc:460] Webserver started at http://127.27.110.193:34541/ using document root <none> and password file <none>
I20260812 07:05:14.522847 22997 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:14.522943 22997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:14.523252 22997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:14.525483 22997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "ba63cfef11da43ac93362cc54d2f817a"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "a4e909c46e8ed9cf8c30a597e441f669"
server_key_iv: "2fbdc7e54944ac81114b12966932bd2a"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.526080 22997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "ba63cfef11da43ac93362cc54d2f817a"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "a4e909c46e8ed9cf8c30a597e441f669"
server_key_iv: "2fbdc7e54944ac81114b12966932bd2a"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.529903 22997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:05:14.532497 23024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.533727 22997 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:05:14.533871 22997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "ba63cfef11da43ac93362cc54d2f817a"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "a4e909c46e8ed9cf8c30a597e441f669"
server_key_iv: "2fbdc7e54944ac81114b12966932bd2a"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.533986 22997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:14.560832 22997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:14.564543 22997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:14.564776 22997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:14.565428 22997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:14.566414 22997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:14.566488 22997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.566558 22997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:14.566612 22997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.576301 22997 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:39599
I20260812 07:05:14.576332 23221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:39599 every 8 connection(s)
I20260812 07:05:14.577258 22997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:05:14.586409 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 22997
I20260812 07:05:14.586547 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:14.586834 28091 external_mini_cluster.cc:1428] Setting key 8ec323ee44a4f3e5a61a8fbdce6bdc43
I20260812 07:05:14.591688 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:14.579193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:41307 (local address 127.27.110.254:42957)
0812 07:05:14.579354 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:14.579358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:14.580436 (+ 1078us) server_negotiation.cc:407] Connection header received
0812 07:05:14.581396 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:14.581400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:14.581459 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:14.581568 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:14.582769 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.583367 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.584541 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.584781 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.584896 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:14.587512 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:14.587516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:14.587519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:14.587548 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:14.589364 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:14.589864 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:14.589870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:14.589875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:14.589973 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:14.590306 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:14.590312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:14.590317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:14.590592 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:14.590712 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:14.591423 (+ 711us) server_negotiation.cc:299] Negotiation successful
0812 07:05:14.591532 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:05:14.592857 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:14.579585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:41307)
0812 07:05:14.580275 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:14.580318 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:14.581181 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:14.581753 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:14.581759 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:14.582126 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:14.582610 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:14.582625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.583502 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.583506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:14.584408 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:14.584418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.584937 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.584941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:14.585118 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:14.585675 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:14.585690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:14.587342 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:14.589488 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:14.589493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:14.589505 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:14.589762 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:14.590068 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:14.590070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:14.590072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:14.590214 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:14.590722 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:14.590731 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:14.591130 (+ 399us) client_negotiation.cc:769] Sending connection context
0812 07:05:14.591361 (+ 231us) client_negotiation.cc:240] Negotiation successful
0812 07:05:14.591733 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":113,"threads_started":1}
I20260812 07:05:14.594292 23222 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42957
I20260812 07:05:14.594599 23222 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:14.595223 23222 heartbeater.cc:511] Master 127.27.110.254:42957 requested a full tablet report, sending...
I20260812 07:05:14.596757 22874 ts_manager.cc:194] Registered new tserver with Master: ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599)
I20260812 07:05:14.597882 22874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:41307
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:05:14.616496 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42957
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:14.732460 23229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:14.732784 23229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:14.732883 23229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:14.737659 23229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:14.737746 23229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:14.737833 23229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:05:14.743355 23229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:14.744405 23229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:14.745451 23229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:14.755306 23237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:14.755434 23229 server_base.cc:1034] running on GCE node
W20260812 07:05:14.755335 23238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:14.755300 23244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:14.755966 23229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:14.756691 23229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:14.764098 23229 hybrid_clock.cc:648] HybridClock initialized: now 1786518314764079 us; error 49 us; skew 500 ppm
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:14.766954 23229 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:05:14.767884 23229 webserver.cc:460] Webserver started at http://127.27.110.194:39663/ using document root <none> and password file <none>
I20260812 07:05:14.768405 23229 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:14.768471 23229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:14.768640 23229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:14.770653 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "bb7d31bddb3d48899dac1796f03c71a3"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "358483acb81df9e38d1cc3cc4ebd3898"
server_key_iv: "8ad4fc4589acf4f24577add87f8baa2d"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.771242 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "bb7d31bddb3d48899dac1796f03c71a3"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "358483acb81df9e38d1cc3cc4ebd3898"
server_key_iv: "8ad4fc4589acf4f24577add87f8baa2d"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.775146 23229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:05:14.777822 23252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.779271 23229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:14.779376 23229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "bb7d31bddb3d48899dac1796f03c71a3"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "358483acb81df9e38d1cc3cc4ebd3898"
server_key_iv: "8ad4fc4589acf4f24577add87f8baa2d"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.779477 23229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:14.797684 23229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:14.801002 23229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:14.801178 23229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:14.801788 23229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:14.802600 23229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:14.802650 23229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.802691 23229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:14.802707 23229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:14.811544 23229 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:42085
I20260812 07:05:14.811569 23451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:42085 every 8 connection(s)
I20260812 07:05:14.812427 23229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:05:14.814482 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 23229
I20260812 07:05:14.814639 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:05:14.814921 28091 external_mini_cluster.cc:1428] Setting key 1faea9869237d3c9a736e9e6649712b2
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:14.828218 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:14.814347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:38797 (local address 127.27.110.254:42957)
0812 07:05:14.814522 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:14.814528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:14.815742 (+ 1214us) server_negotiation.cc:407] Connection header received
0812 07:05:14.816607 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:14.816611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:14.816670 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:14.816776 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:14.818074 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.818728 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.820725 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.820951 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.821054 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:14.824063 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:14.824070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:14.824076 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:14.824111 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:14.825696 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:14.826403 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:14.826405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:14.826407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:14.826481 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:14.826902 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:14.826905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:14.826906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:14.827150 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:14.827231 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:14.827912 (+ 681us) server_negotiation.cc:299] Negotiation successful
0812 07:05:14.828020 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:05:14.829480 23464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:14.814796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:38797)
0812 07:05:14.815570 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:14.815623 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:14.816407 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:14.816934 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:14.816941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:14.817438 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:14.817921 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:14.817935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.819092 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.819097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:14.820559 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:14.820576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:14.821125 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:14.821132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:14.821379 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:14.822032 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:14.822049 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:14.823847 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:14.825827 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:14.825838 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:14.825856 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:14.826288 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:14.826582 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:14.826588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:14.826591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:14.826798 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:14.827260 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:14.827268 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:14.827613 (+ 345us) client_negotiation.cc:769] Sending connection context
0812 07:05:14.827861 (+ 248us) client_negotiation.cc:240] Negotiation successful
0812 07:05:14.828312 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":168,"threads_started":1}
I20260812 07:05:14.830798 23455 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42957
I20260812 07:05:14.831039 23455 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:14.831609 23455 heartbeater.cc:511] Master 127.27.110.254:42957 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:05:14.832698 22874 ts_manager.cc:194] Registered new tserver with Master: bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
I20260812 07:05:14.833308 22874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:38797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:05:14.848071 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42957
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:14.962532 23476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:14.962793 23476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:14.962878 23476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:14.966843 23476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:14.966902 23476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:14.967007 23476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:05:14.972185 23476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:14.973254 23476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:14.974392 23476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:14.984298 23487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:14.984253 23490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:14.984310 23486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:14.984695 23476 server_base.cc:1034] running on GCE node
I20260812 07:05:14.985088 23476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:14.985719 23476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:14.986910 23476 hybrid_clock.cc:648] HybridClock initialized: now 1786518314986895 us; error 27 us; skew 500 ppm
Aug 12 07:05:14 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:14.989656 23476 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:05:14.990566 23476 webserver.cc:460] Webserver started at http://127.27.110.195:42387/ using document root <none> and password file <none>
I20260812 07:05:14.991197 23476 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:14.991292 23476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:14.991525 23476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:05:14.993561 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "b8cb6174519c4084a53039c198d3570e"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "63c9a3c5eee74820e37c2fd5e4441e08"
server_key_iv: "3efba96ff7d58c4e7838b6d2cab3c3bc"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.994246 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "b8cb6174519c4084a53039c198d3570e"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "63c9a3c5eee74820e37c2fd5e4441e08"
server_key_iv: "3efba96ff7d58c4e7838b6d2cab3c3bc"
server_key_version: "encryptionkey@0"
I20260812 07:05:14.998050 23476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 07:05:15.000490 23500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:15.001806 23476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:05:15.001943 23476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "b8cb6174519c4084a53039c198d3570e"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "63c9a3c5eee74820e37c2fd5e4441e08"
server_key_iv: "3efba96ff7d58c4e7838b6d2cab3c3bc"
server_key_version: "encryptionkey@0"
I20260812 07:05:15.002058 23476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:15.014101 23476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:15.017170 23476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:15.017393 23476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:15.017997 23476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:15.018833 23476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:15.018906 23476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:15.018977 23476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:15.019029 23476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:15.028666 23476 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:38629
I20260812 07:05:15.028728 23704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:38629 every 8 connection(s)
I20260812 07:05:15.029578 23476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:05:15.035437 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 23476
I20260812 07:05:15.035584 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:05:15.035945 28091 external_mini_cluster.cc:1428] Setting key 49e389efc4cd620ac95605ffce6e3422
Aug 12 07:05:15 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518314, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:15.043915 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.031467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:49273 (local address 127.27.110.254:42957)
0812 07:05:15.031616 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.031620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.032749 (+ 1129us) server_negotiation.cc:407] Connection header received
0812 07:05:15.033615 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.033618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.033669 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.033753 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:15.034935 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.035701 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.037220 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.037494 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.037620 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.040312 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:15.040316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:15.040318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:15.040354 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:15.041851 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:15.042360 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:15.042364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:15.042366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:15.042446 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:15.042806 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:15.042810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:15.042812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:15.042993 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:15.043130 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.043629 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.043748 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:05:15.044761 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.031868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:49273)
0812 07:05:15.032585 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.032630 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:15.033400 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:15.033923 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:15.033930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:15.034304 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:15.034782 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.034796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.035845 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.035848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.037080 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.037092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.037678 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.037682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.037901 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.038602 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:15.038618 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:15.040150 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:15.041977 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:15.041984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:15.041999 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:15.042256 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:15.042550 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:15.042553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:15.042555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:15.042711 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:15.043135 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:15.043143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:15.043386 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 07:05:15.043586 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 07:05:15.043907 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":112,"threads_started":1}
I20260812 07:05:15.045857 23705 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42957
I20260812 07:05:15.046145 23705 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:15.046663 23705 heartbeater.cc:511] Master 127.27.110.254:42957 requested a full tablet report, sending...
I20260812 07:05:15.048054 22874 ts_manager.cc:194] Registered new tserver with Master: b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629)
I20260812 07:05:15.048656 22874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:49273
I20260812 07:05:15.050338 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:05:15.060613 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.052617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.27.110.254:42957)
0812 07:05:15.052812 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.052817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.052970 (+ 153us) server_negotiation.cc:407] Connection header received
0812 07:05:15.053146 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.053150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.053198 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.053285 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:15.054013 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.054526 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.055512 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.055747 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.055858 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.057271 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:15.057275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:15.057278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:15.057309 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:15.058756 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:15.059267 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:15.059271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:15.059272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:15.059356 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:15.059661 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:15.059664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:15.059666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:15.059887 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:15.060038 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.060317 (+ 279us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.060437 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 07:05:15.063709 22874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:05:15.066783 22874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:05:15.081602 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.075525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38629 (local address 127.0.0.1:50458)
0812 07:05:15.076968 (+ 1443us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.076981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:15.077156 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:15.077725 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:15.077728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:15.077748 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:15.077834 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.077841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.079306 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.079312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.080092 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.080101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.081268 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.081275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.081361 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.081380 (+ 19us) client_negotiation.cc:769] Sending connection context
0812 07:05:15.081451 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 07:05:15.081508 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1390,"thread_start_us":97,"threads_started":1}
I20260812 07:05:15.081769 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.076004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.27.110.195:38629)
0812 07:05:15.076503 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.076515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.077084 (+ 569us) server_negotiation.cc:407] Connection header received
0812 07:05:15.077311 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.077316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.077571 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.077750 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:15.077930 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.079142 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.080220 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.081068 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.081192 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.081315 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.081503 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.081621 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":107,"threads_started":1}
I20260812 07:05:15.081935 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.075840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.27.110.193:39599)
0812 07:05:15.076265 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.076280 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.076771 (+ 491us) server_negotiation.cc:407] Connection header received
0812 07:05:15.076946 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.076954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.077173 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.077343 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:15.077515 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.078623 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.080310 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.080884 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.081039 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.081183 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.081683 (+ 500us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.081796 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":121,"threads_started":1}
I20260812 07:05:15.082175 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.075345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41358 (local address 127.27.110.194:42085)
0812 07:05:15.076465 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.076479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.076505 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:05:15.076583 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.076587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.076745 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.076911 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:15.077222 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.078446 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.080296 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.081069 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.081192 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.081314 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.081907 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.082014 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":106,"threads_started":1}
I20260812 07:05:15.081811 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.075177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42085 (local address 127.0.0.1:41358)
0812 07:05:15.076096 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.076150 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:15.076282 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:15.076902 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:15.076909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:15.076933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:15.077091 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.077099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.078591 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.078594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.080193 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.080204 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.081392 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.081395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.081502 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.081535 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 07:05:15.081608 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 07:05:15.081676 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":341,"thread_start_us":77,"threads_started":1}
I20260812 07:05:15.081604 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.075247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39599 (local address 127.0.0.1:43212)
0812 07:05:15.076660 (+ 1413us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.076671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:15.076783 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:15.077307 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:15.077309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:15.077325 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:15.077413 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.077418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.078750 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.078754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.080151 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.080164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.081025 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.081028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.081155 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.081196 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:05:15.081304 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 07:05:15.081428 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1351,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260812 07:05:15.085295 23111 tablet_service.cc:1467] Processing CreateTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c (DEFAULT_TABLE table=test-table [id=102e4aec24864318ae41e85d14950c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:15.085318 23586 tablet_service.cc:1467] Processing CreateTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c (DEFAULT_TABLE table=test-table [id=102e4aec24864318ae41e85d14950c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:15.085317 23337 tablet_service.cc:1467] Processing CreateTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c (DEFAULT_TABLE table=test-table [id=102e4aec24864318ae41e85d14950c00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:15.086408 23586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d338b8e18e5b4ac3a894e8bc4e267f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:15.086404 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d338b8e18e5b4ac3a894e8bc4e267f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:15.086405 23111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d338b8e18e5b4ac3a894e8bc4e267f9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:15.092799 23731 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Bootstrap starting.
I20260812 07:05:15.094655 23738 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Bootstrap starting.
I20260812 07:05:15.095405 23733 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Bootstrap starting.
I20260812 07:05:15.095450 23731 tablet_bootstrap.cc:654] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:15.096212 23731 log.cc:824] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:15.096894 23738 tablet_bootstrap.cc:654] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:15.097914 23733 tablet_bootstrap.cc:654] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:15.098258 23738 log.cc:824] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:15.098543 23731 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: No bootstrap required, opened a new log
I20260812 07:05:15.098723 23733 log.cc:824] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:15.098762 23731 ts_tablet_manager.cc:1397] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:05:15.100353 23738 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: No bootstrap required, opened a new log
I20260812 07:05:15.100613 23738 ts_tablet_manager.cc:1397] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260812 07:05:15.100672 23733 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: No bootstrap required, opened a new log
I20260812 07:05:15.100848 23733 ts_tablet_manager.cc:1397] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260812 07:05:15.102429 23731 raft_consensus.cc:348] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.102691 23731 raft_consensus.cc:374] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:15.102746 23731 raft_consensus.cc:729] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba63cfef11da43ac93362cc54d2f817a, State: Initialized, Role: FOLLOWER
I20260812 07:05:15.103268 23731 consensus_queue.cc:260] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.104405 23731 ts_tablet_manager.cc:1428] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:05:15.104610 23222 heartbeater.cc:503] Master 127.27.110.254:42957 was elected leader, sending a full tablet report...
I20260812 07:05:15.104722 23733 raft_consensus.cc:348] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.105022 23733 raft_consensus.cc:374] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:15.105077 23733 raft_consensus.cc:729] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7d31bddb3d48899dac1796f03c71a3, State: Initialized, Role: FOLLOWER
I20260812 07:05:15.105533 23733 consensus_queue.cc:260] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.106057 23738 raft_consensus.cc:348] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.106326 23738 raft_consensus.cc:374] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:15.106375 23738 raft_consensus.cc:729] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8cb6174519c4084a53039c198d3570e, State: Initialized, Role: FOLLOWER
I20260812 07:05:15.106381 23455 heartbeater.cc:503] Master 127.27.110.254:42957 was elected leader, sending a full tablet report...
I20260812 07:05:15.106659 23733 ts_tablet_manager.cc:1428] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260812 07:05:15.106997 23738 consensus_queue.cc:260] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.107754 23705 heartbeater.cc:503] Master 127.27.110.254:42957 was elected leader, sending a full tablet report...
I20260812 07:05:15.108147 23738 ts_tablet_manager.cc:1428] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
W20260812 07:05:15.280537 23706 tablet.cc:2369] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:05:15.288025 23744 raft_consensus.cc:482] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:15.288278 23744 raft_consensus.cc:504] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.289556 23744 leader_election.cc:288] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085), b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629)
I20260812 07:05:15.294955 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.289884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38629 (local address 127.27.110.193:35881)
0812 07:05:15.290309 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.290323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:15.290415 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:15.290758 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:15.290760 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:15.290772 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:15.291115 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.291123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.292346 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.292349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.293712 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.293724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.294446 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.294449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.294581 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.294604 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:05:15.294704 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 07:05:15.294782 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":100,"thread_start_us":127,"threads_started":1}
I20260812 07:05:15.294927 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.289995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:35881 (local address 127.27.110.195:38629)
0812 07:05:15.290131 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.290136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.290431 (+ 295us) server_negotiation.cc:407] Connection header received
0812 07:05:15.290583 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.290588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.290650 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.290728 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:15.291506 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.292243 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.293881 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.294321 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.294423 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.294466 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.294724 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.294802 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:05:15.295811 23723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.289895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:54503 (local address 127.27.110.194:42085)
0812 07:05:15.290028 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.290034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.290338 (+ 304us) server_negotiation.cc:407] Connection header received
0812 07:05:15.290493 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.290498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.290565 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.290664 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:15.291257 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.292118 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.294559 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.294998 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.295165 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.295217 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.295638 (+ 421us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.295719 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:05:15.295835 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.289780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42085 (local address 127.27.110.193:54503)
0812 07:05:15.290196 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.290213 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:15.290361 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:15.290731 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:15.290736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:15.290761 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:15.291115 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.291123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.292242 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.292246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.293712 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:15.293724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.295353 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.295358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:15.295485 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.295526 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 07:05:15.295598 (+ 72us) client_negotiation.cc:240] Negotiation successful
0812 07:05:15.295681 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":132,"thread_start_us":118,"threads_started":1}
I20260812 07:05:15.296026 23617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338b8e18e5b4ac3a894e8bc4e267f9c" candidate_uuid: "ba63cfef11da43ac93362cc54d2f817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8cb6174519c4084a53039c198d3570e" is_pre_election: true
I20260812 07:05:15.296425 23617 raft_consensus.cc:2454] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba63cfef11da43ac93362cc54d2f817a in term 0.
I20260812 07:05:15.296597 23372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338b8e18e5b4ac3a894e8bc4e267f9c" candidate_uuid: "ba63cfef11da43ac93362cc54d2f817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7d31bddb3d48899dac1796f03c71a3" is_pre_election: true
I20260812 07:05:15.296912 23372 raft_consensus.cc:2454] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba63cfef11da43ac93362cc54d2f817a in term 0.
I20260812 07:05:15.297037 23033 leader_election.cc:302] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cb6174519c4084a53039c198d3570e, ba63cfef11da43ac93362cc54d2f817a; no voters:
I20260812 07:05:15.297341 23744 raft_consensus.cc:2781] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:05:15.297425 23744 raft_consensus.cc:482] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:05:15.297482 23744 raft_consensus.cc:3037] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:15.298563 23744 raft_consensus.cc:504] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.298930 23744 leader_election.cc:288] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 1 election: Requested vote from peers bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085), b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629)
I20260812 07:05:15.299422 23372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338b8e18e5b4ac3a894e8bc4e267f9c" candidate_uuid: "ba63cfef11da43ac93362cc54d2f817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7d31bddb3d48899dac1796f03c71a3"
I20260812 07:05:15.299573 23372 raft_consensus.cc:3037] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:15.299702 23617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d338b8e18e5b4ac3a894e8bc4e267f9c" candidate_uuid: "ba63cfef11da43ac93362cc54d2f817a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8cb6174519c4084a53039c198d3570e"
I20260812 07:05:15.299896 23617 raft_consensus.cc:3037] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:15.300804 23372 raft_consensus.cc:2454] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba63cfef11da43ac93362cc54d2f817a in term 1.
I20260812 07:05:15.301025 23617 raft_consensus.cc:2454] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba63cfef11da43ac93362cc54d2f817a in term 1.
I20260812 07:05:15.301154 23029 leader_election.cc:302] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba63cfef11da43ac93362cc54d2f817a, bb7d31bddb3d48899dac1796f03c71a3; no voters:
I20260812 07:05:15.301398 23744 raft_consensus.cc:2781] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:15.301625 23744 raft_consensus.cc:686] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 LEADER]: Becoming Leader. State: Replica: ba63cfef11da43ac93362cc54d2f817a, State: Running, Role: LEADER
I20260812 07:05:15.301965 23744 consensus_queue.cc:237] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:15.305334 22873 catalog_manager.cc:5495] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a reported cstate change: term changed from 0 to 1, leader changed from <none> to ba63cfef11da43ac93362cc54d2f817a (127.27.110.193). New cstate: current_term: 1 leader_uuid: "ba63cfef11da43ac93362cc54d2f817a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } health_report { overall_health: UNKNOWN } } }
W20260812 07:05:15.313622 23459 tablet.cc:2369] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:05:15.328732 23223 tablet.cc:2369] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:05:15.372565 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.369121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.27.110.193:39599)
0812 07:05:15.369266 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:15.369269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:15.369284 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:05:15.369422 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:15.369425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:15.369483 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:15.369583 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:05:15.369806 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.370310 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.371202 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:15.371453 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:15.371539 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:15.371822 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:05:15.372257 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:05:15.372350 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:15.372400 (+ 50us) server_negotiation.cc:299] Negotiation successful
0812 07:05:15.372451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:05:15.378522 23617 raft_consensus.cc:1261] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Refusing update from remote peer ba63cfef11da43ac93362cc54d2f817a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:05:15.378522 23372 raft_consensus.cc:1261] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Refusing update from remote peer ba63cfef11da43ac93362cc54d2f817a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:05:15.379271 23744 consensus_queue.cc:1035] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:05:15.379477 23762 consensus_queue.cc:1035] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:05:15.388572 23768 mvcc.cc:205] Tried to move back new op lower bound from 7317579019782504448 to 7317579019479699456. Current Snapshot: MvccSnapshot[applied={T|T < 7317579019782504448 or (T in {7317579019782504448})}]
I20260812 07:05:15.388962 23769 mvcc.cc:205] Tried to move back new op lower bound from 7317579019782504448 to 7317579019479699456. Current Snapshot: MvccSnapshot[applied={T|T < 7317579019782504448}]
I20260812 07:05:15.396275 22873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:05:15.396467 22873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:05:15.399128 22873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Sending DeleteTablet for 3 replicas of tablet d338b8e18e5b4ac3a894e8bc4e267f9c
I20260812 07:05:15.400000 23111 tablet_service.cc:1514] Processing DeleteTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:43212
I20260812 07:05:15.400144 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 22997
I20260812 07:05:15.400066 23337 tablet_service.cc:1514] Processing DeleteTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:41358
I20260812 07:05:15.405776 23586 tablet_service.cc:1514] Processing DeleteTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:50458
I20260812 07:05:15.407317 23780 tablet_replica.cc:330] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: stopping tablet replica
I20260812 07:05:15.407531 23779 tablet_replica.cc:330] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: stopping tablet replica
I20260812 07:05:15.407752 23780 raft_consensus.cc:2229] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:15.407850 23779 raft_consensus.cc:2229] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:15.408144 23779 raft_consensus.cc:2258] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:15.408156 23780 raft_consensus.cc:2258] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:15.411015 23780 ts_tablet_manager.cc:1905] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:15.411427 23779 ts_tablet_manager.cc:1905] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:15.413309 23780 ts_tablet_manager.cc:1918] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:15.413437 23780 log.cc:1196] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/d338b8e18e5b4ac3a894e8bc4e267f9c
I20260812 07:05:15.413749 23780 ts_tablet_manager.cc:1939] T d338b8e18e5b4ac3a894e8bc4e267f9c P b8cb6174519c4084a53039c198d3570e: Deleting consensus metadata
I20260812 07:05:15.414956 23779 ts_tablet_manager.cc:1918] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:15.414987 22852 catalog_manager.cc:4841] TS b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): tablet d338b8e18e5b4ac3a894e8bc4e267f9c (table test-table [id=102e4aec24864318ae41e85d14950c00]) successfully deleted
I20260812 07:05:15.415093 23779 log.cc:1196] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/d338b8e18e5b4ac3a894e8bc4e267f9c
I20260812 07:05:15.415445 23779 ts_tablet_manager.cc:1939] T d338b8e18e5b4ac3a894e8bc4e267f9c P bb7d31bddb3d48899dac1796f03c71a3: Deleting consensus metadata
I20260812 07:05:15.416787 22846 catalog_manager.cc:4841] TS bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): tablet d338b8e18e5b4ac3a894e8bc4e267f9c (table test-table [id=102e4aec24864318ae41e85d14950c00]) successfully deleted
W20260812 07:05:15.418218 22847 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:39599 (error 108)
I20260812 07:05:15.418525 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 23229
I20260812 07:05:15.419412 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:15.419000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39599 (local address 127.0.0.1:57676)
0812 07:05:15.419210 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:15.419298 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:39599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260812 07:05:15.419652 22847 catalog_manager.cc:4584] TS ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d338b8e18e5b4ac3a894e8bc4e267f9c: Network error: Client connection negotiation failed: client connection to 127.27.110.193:39599: connect: Connection refused (error 111)
I20260812 07:05:15.437295 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 23476
I20260812 07:05:15.452886 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 22821
I20260812 07:05:35.466508 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42957
--webserver_interface=127.27.110.254
--webserver_port=36065
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:35.584077 23784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:35.584393 23784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:35.584478 23784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:35.588687 23784 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 07:05:35.588771 23784 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 07:05:35.588800 23784 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 07:05:35.588831 23784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:05:35.588872 23784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:35.588918 23784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:35.588954 23784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:05:35.588985 23784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:05:35.594230 23784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:35.595342 23784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:35.596395 23784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:35.610780 23798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:35.610781 23801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:35.610780 23797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:35.611164 23784 server_base.cc:1034] running on GCE node
I20260812 07:05:35.611934 23784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:35.613129 23784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:35.614339 23784 hybrid_clock.cc:648] HybridClock initialized: now 1786518335614303 us; error 57 us; skew 500 ppm
Aug 12 07:05:35 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:35.621711 23784 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:05:35.622759 23784 webserver.cc:460] Webserver started at http://127.27.110.254:36065/ using document root <none> and password file <none>
I20260812 07:05:35.623437 23784 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:35.623555 23784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:35.627674 23784 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:05:35.630010 23818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:35.631438 23784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:35.631584 23784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "572fb8ecc78a49e2b08fd6c24f9439d1"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "ef9e2f0be3df293c8815396cf4fe0043"
server_key_iv: "3afd6c2b1c963bae77d4c5674a6ff9ce"
server_key_version: "encryptionkey@0"
I20260812 07:05:35.632256 23784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:35.657943 23784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:35.661538 23784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:35.661768 23784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:35.669720 23784 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42957
I20260812 07:05:35.670027 23906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42957 every 8 connection(s)
I20260812 07:05:35.670917 23784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:05:35.675560 23910 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:05:35.675885 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 23784
I20260812 07:05:35.676379 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:39599
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:42957
--webserver_port=34541
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:05:35.677974 23910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap starting.
I20260812 07:05:35.702905 23910 log.cc:824] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:35.714525 23910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:05:35.715011 23910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap complete.
I20260812 07:05:35.720795 23910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:35.721340 23910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572fb8ecc78a49e2b08fd6c24f9439d1, State: Initialized, Role: FOLLOWER
I20260812 07:05:35.721920 23910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:35.722076 23910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:05:35.722154 23910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:05:35.722254 23910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 1 FOLLOWER]: Advancing to term 2
I20260812 07:05:35.725080 23910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:35.725466 23910 leader_election.cc:302] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 572fb8ecc78a49e2b08fd6c24f9439d1; no voters:
I20260812 07:05:35.725836 23910 leader_election.cc:288] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 07:05:35.726001 23917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 07:05:35.726298 23917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 LEADER]: Becoming Leader. State: Replica: 572fb8ecc78a49e2b08fd6c24f9439d1, State: Running, Role: LEADER
I20260812 07:05:35.726722 23917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:35.727286 23910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:05:35.728446 23918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } } }
I20260812 07:05:35.728446 23919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 572fb8ecc78a49e2b08fd6c24f9439d1. Latest consensus state: current_term: 2 leader_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } } }
I20260812 07:05:35.728631 23918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:35.728631 23919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:35.733206 23946 catalog_manager.cc:1259] Loaded cluster ID: 4f95bdc1595f4b8b81a2271d12462696
I20260812 07:05:35.733278 23946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: loading cluster ID for follower catalog manager: success
I20260812 07:05:35.735885 23946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: acquiring CA information for follower catalog manager: success
I20260812 07:05:35.737674 23946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:05:35.737841 23940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:05:35.740973 23940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:05:35.741482 23940 catalog_manager.cc:1259] Loaded cluster ID: 4f95bdc1595f4b8b81a2271d12462696
I20260812 07:05:35.741556 23940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:05:35.742264 23940 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:05:35.742784 23940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Loaded TSK: 0
I20260812 07:05:35.758239 23940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Generated new TSK 1
I20260812 07:05:35.758513 23940 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:05:35.816957 23913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:35.817334 23913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:35.817435 23913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:35.821810 23913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:35.821904 23913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:35.821995 23913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:05:35.827654 23913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=34541
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:35.828856 23913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:35.829921 23913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:35.840077 23964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:35.840145 23967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:35.840086 23960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:35.840487 23913 server_base.cc:1034] running on GCE node
I20260812 07:05:35.840955 23913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:35.841750 23913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:35.842962 23913 hybrid_clock.cc:648] HybridClock initialized: now 1786518335842935 us; error 42 us; skew 500 ppm
Aug 12 07:05:35 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:35.846544 23913 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:05:35.847743 23913 webserver.cc:460] Webserver started at http://127.27.110.193:34541/ using document root <none> and password file <none>
I20260812 07:05:35.848336 23913 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:35.848464 23913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:35.853091 23913 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:35.855623 23986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:35.857035 23913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:35.857197 23913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "ba63cfef11da43ac93362cc54d2f817a"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "a4e909c46e8ed9cf8c30a597e441f669"
server_key_iv: "2fbdc7e54944ac81114b12966932bd2a"
server_key_version: "encryptionkey@0"
I20260812 07:05:35.857771 23913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:35.871935 23913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:35.875483 23913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:35.875712 23913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:35.876338 23913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:35.877549 23998 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 07:05:35.881726 23913 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 07:05:35.881794 23913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260812 07:05:35.881843 23913 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 07:05:35.883677 23913 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 07:05:35.883738 23913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260812 07:05:35.883996 23998 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Bootstrap starting.
I20260812 07:05:35.895253 23913 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:39599
I20260812 07:05:35.895316 24180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:39599 every 8 connection(s)
I20260812 07:05:35.896538 23913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:05:35.905310 23998 log.cc:824] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:35.906364 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 23913
I20260812 07:05:35.906821 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:42085
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:42957
--webserver_port=39663
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 07:05:35 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:35.914318 23998 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:05:35.914763 23998 tablet_bootstrap.cc:492] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Bootstrap complete.
I20260812 07:05:35.914809 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:35.898869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:40481)
0812 07:05:35.899511 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:35.899555 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:35.900384 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:35.901511 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:35.901519 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:35.901980 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:35.902444 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:35.902458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.903863 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.903866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:35.904720 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:35.904730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.905455 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.905459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:35.905665 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:35.906395 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:35.906416 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:35.908290 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:35.911159 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:35.911165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:35.911179 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:35.911548 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:35.911981 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:35.911984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:35.911986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:35.912164 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:35.912990 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:35.912996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:35.913296 (+ 300us) client_negotiation.cc:769] Sending connection context
0812 07:05:35.913481 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 07:05:35.913801 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":208,"threads_started":1}
I20260812 07:05:35.916322 23998 ts_tablet_manager.cc:1397] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260812 07:05:35.916709 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:35.898891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:40481 (local address 127.27.110.254:42957)
0812 07:05:35.899312 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:35.899332 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:35.899684 (+ 352us) server_negotiation.cc:407] Connection header received
0812 07:05:35.900811 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:35.900822 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:35.901119 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:35.901456 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:35.902639 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.903694 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.904847 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.905310 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.905498 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:35.908518 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:35.908526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:35.908548 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:35.908590 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:35.911014 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:35.911683 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:35.911692 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:35.911701 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:35.911865 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:35.912273 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:35.912279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:35.912284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:35.912857 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:35.913001 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:35.915534 (+ 2533us) server_negotiation.cc:299] Negotiation successful
0812 07:05:35.915856 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":180,"threads_started":1}
I20260812 07:05:35.919014 24181 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42957
I20260812 07:05:35.919318 24181 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:35.919140 23998 raft_consensus.cc:348] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:35.919659 23998 raft_consensus.cc:729] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba63cfef11da43ac93362cc54d2f817a, State: Initialized, Role: FOLLOWER
I20260812 07:05:35.919965 24181 heartbeater.cc:511] Master 127.27.110.254:42957 requested a full tablet report, sending...
I20260812 07:05:35.920185 23998 consensus_queue.cc:260] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } } peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } }
I20260812 07:05:35.921176 23998 ts_tablet_manager.cc:1428] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:05:35.922761 23849 ts_manager.cc:194] Registered new tserver with Master: ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599)
I20260812 07:05:35.926255 23849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:40481
I20260812 07:05:35.929054 24181 heartbeater.cc:503] Master 127.27.110.254:42957 was elected leader, sending a full tablet report...
Aug 12 07:05:35 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518335, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.193@KRBTEST.COM
I20260812 07:05:35.936625 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:35.925641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52562 (local address 127.27.110.193:39599)
0812 07:05:35.926258 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:35.926273 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:35.926300 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:05:35.926522 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:35.926527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:35.926721 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:35.926902 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:35.927771 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.928718 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.929901 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.930245 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.930365 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:35.932731 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:35.932743 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:35.932759 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:35.932782 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:35.934442 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:35.935012 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:35.935016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:35.935018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:35.935181 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:35.935517 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:35.935523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:35.935527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:35.935789 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:35.935936 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:35.936301 (+ 365us) server_negotiation.cc:299] Negotiation successful
0812 07:05:35.936465 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":112,"threads_started":1}
I20260812 07:05:35.936635 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:35.925547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39599 (local address 127.0.0.1:52562)
0812 07:05:35.926141 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:35.926191 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:35.926315 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:35.926896 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:35.926911 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:35.927537 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:35.927645 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:35.927652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.928828 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.928833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:35.929790 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:35.929801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:35.930368 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:35.930372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:35.930499 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:35.931157 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:35.931178 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:35.932587 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:35.934571 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:35.934618 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:35.934624 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:35.934868 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:35.935287 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:35.935290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:35.935292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:35.935392 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:35.935912 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:35.935918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:35.936051 (+ 133us) client_negotiation.cc:769] Sending connection context
0812 07:05:35.936211 (+ 160us) client_negotiation.cc:240] Negotiation successful
0812 07:05:35.936427 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":110,"threads_started":1}
I20260812 07:05:35.937779 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet d338b8e18e5b4ac3a894e8bc4e267f9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:52562
I20260812 07:05:35.938695 24203 tablet_replica.cc:330] stopping tablet replica
I20260812 07:05:35.939043 24203 raft_consensus.cc:2229] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:35.939333 24203 raft_consensus.cc:2258] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:35.940469 24203 ts_tablet_manager.cc:1905] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:35.945079 24203 ts_tablet_manager.cc:1918] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:35.945186 24203 log.cc:1196] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal/wals/d338b8e18e5b4ac3a894e8bc4e267f9c
I20260812 07:05:35.945465 24203 ts_tablet_manager.cc:1939] T d338b8e18e5b4ac3a894e8bc4e267f9c P ba63cfef11da43ac93362cc54d2f817a: Deleting consensus metadata
I20260812 07:05:35.947855 23823 catalog_manager.cc:4841] TS ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599): tablet d338b8e18e5b4ac3a894e8bc4e267f9c (table test-table [id=102e4aec24864318ae41e85d14950c00]) successfully deleted
W20260812 07:05:36.034299 24193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:36.034607 24193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:36.034690 24193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:36.038945 24193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:36.039039 24193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:36.039141 24193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:05:36.044432 24193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:42085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=39663
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:36.045539 24193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:36.046593 24193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:36.056437 24217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:36.056470 24212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:36.056445 24210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:36.056785 24193 server_base.cc:1034] running on GCE node
I20260812 07:05:36.057164 24193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:36.057801 24193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:36.059003 24193 hybrid_clock.cc:648] HybridClock initialized: now 1786518336058970 us; error 44 us; skew 500 ppm
Aug 12 07:05:36 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:36.062160 24193 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:05:36.062973 24193 webserver.cc:460] Webserver started at http://127.27.110.194:39663/ using document root <none> and password file <none>
I20260812 07:05:36.063566 24193 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:36.063642 24193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:36.067493 24193 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:36.069614 24226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:36.071000 24193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:05:36.071236 24193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "bb7d31bddb3d48899dac1796f03c71a3"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "358483acb81df9e38d1cc3cc4ebd3898"
server_key_iv: "8ad4fc4589acf4f24577add87f8baa2d"
server_key_version: "encryptionkey@0"
I20260812 07:05:36.071803 24193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:36.090130 24193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:36.093508 24193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:36.093761 24193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:36.094451 24193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:36.095461 24193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:36.095512 24193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:36.095597 24193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:36.095642 24193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:36.105481 24193 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:42085
I20260812 07:05:36.105515 24426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:42085 every 8 connection(s)
I20260812 07:05:36.106573 24193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:05:36.116611 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 24193
Aug 12 07:05:36 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:36.117151 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:38629
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:42957
--webserver_port=42387
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:05:36.122339 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.108457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:40297 (local address 127.27.110.254:42957)
0812 07:05:36.108606 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.108610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.109902 (+ 1292us) server_negotiation.cc:407] Connection header received
0812 07:05:36.110920 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.110924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.110995 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.111137 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:36.112462 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.113066 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.114326 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.114604 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.114743 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.117717 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:36.117724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:36.117730 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:36.117769 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:36.119710 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.120478 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.120481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.120483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.120570 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.121023 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.121026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.121028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.121228 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:36.121361 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.121970 (+ 609us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.122145 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 07:05:36.123593 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.108927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:40297)
0812 07:05:36.109717 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.109764 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:36.110690 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:36.111344 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:36.111351 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:36.111759 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:36.112290 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.112304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.113218 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.113226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.114184 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.114194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.114787 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.114793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.114980 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.115700 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:36.115718 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:36.117532 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:36.119866 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:36.119877 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:36.119896 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:36.120338 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:36.120683 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:36.120688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:36.120692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:36.120913 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:36.121396 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:36.121405 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:36.121700 (+ 295us) client_negotiation.cc:769] Sending connection context
0812 07:05:36.121921 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 07:05:36.122292 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":128,"threads_started":1}
I20260812 07:05:36.124960 24427 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42957
I20260812 07:05:36.125242 24427 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:36.125800 24427 heartbeater.cc:511] Master 127.27.110.254:42957 requested a full tablet report, sending...
I20260812 07:05:36.127039 23849 ts_manager.cc:194] Registered new tserver with Master: bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
I20260812 07:05:36.127974 23849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:40297
W20260812 07:05:36.236517 24433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:36.236793 24433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:36.236838 24433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:36.240998 24433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:36.241076 24433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:36.241138 24433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:05:36.246574 24433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:38629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=42387
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:36.247784 24433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:36.248800 24433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:36.259009 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:36.259009 24445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:36.259095 24433 server_base.cc:1034] running on GCE node
W20260812 07:05:36.258991 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:36.259783 24433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:36.260483 24433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:36.261699 24433 hybrid_clock.cc:648] HybridClock initialized: now 1786518336261681 us; error 30 us; skew 500 ppm
Aug 12 07:05:36 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:36.264664 24433 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:05:36.265614 24433 webserver.cc:460] Webserver started at http://127.27.110.195:42387/ using document root <none> and password file <none>
I20260812 07:05:36.266193 24433 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:36.266304 24433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:36.270825 24433 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:36.273782 24463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:36.275470 24433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:36.275633 24433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "b8cb6174519c4084a53039c198d3570e"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "63c9a3c5eee74820e37c2fd5e4441e08"
server_key_iv: "3efba96ff7d58c4e7838b6d2cab3c3bc"
server_key_version: "encryptionkey@0"
I20260812 07:05:36.276245 24433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:36.301674 24433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:36.305068 24433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:36.305282 24433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:36.305935 24433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:36.306821 24433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:36.306893 24433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:36.306969 24433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:36.307013 24433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:36.316632 24433 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:38629
I20260812 07:05:36.316654 24653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:38629 every 8 connection(s)
I20260812 07:05:36.317675 24433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:05:36.326946 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 24433
Aug 12 07:05:36 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:05:36.334444 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.319671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:59999 (local address 127.27.110.254:42957)
0812 07:05:36.319831 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.319835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.320933 (+ 1098us) server_negotiation.cc:407] Connection header received
0812 07:05:36.321962 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.321967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.322036 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.322133 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:36.323754 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.324579 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.326688 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.327179 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.327308 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.330634 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:36.330640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:36.330643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:36.330675 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:36.332282 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.332813 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.332817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.332819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.332906 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.333282 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.333285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.333287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.333464 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:36.333585 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.334101 (+ 516us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.334261 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 07:05:36.335376 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.320008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:59999)
0812 07:05:36.320749 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.320798 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:36.321699 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:36.322348 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:36.322355 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:36.322764 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:36.323444 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.323472 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.324744 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.324748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.326536 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.326548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.327357 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.327362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.327620 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.328447 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:36.328470 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:36.330419 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:36.332411 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:36.332417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:36.332433 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:36.332707 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:36.333003 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:36.333006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:36.333008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:36.333179 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:36.333583 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:36.333590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:36.333848 (+ 258us) client_negotiation.cc:769] Sending connection context
0812 07:05:36.334054 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 07:05:36.334398 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":116,"threads_started":1}
I20260812 07:05:36.336668 24657 heartbeater.cc:347] Connected to a master server at 127.27.110.254:42957
I20260812 07:05:36.336928 24657 heartbeater.cc:464] Registering TS with master...
I20260812 07:05:36.337448 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.327719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.27.110.254:42957)
0812 07:05:36.327999 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.328003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.328616 (+ 613us) server_negotiation.cc:407] Connection header received
0812 07:05:36.328783 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.328786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.328828 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.328915 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:36.329962 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.330555 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.331704 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.331968 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.332092 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.333549 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:36.333554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:36.333556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:36.333583 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:36.335509 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.336082 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.336085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.336086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.336215 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.336551 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.336555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.336557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.336751 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:36.336982 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.337139 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.337271 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260812 07:05:36.337529 24657 heartbeater.cc:511] Master 127.27.110.254:42957 requested a full tablet report, sending...
I20260812 07:05:36.338676 23849 ts_manager.cc:194] Registered new tserver with Master: b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629)
I20260812 07:05:36.339452 23849 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:59999
I20260812 07:05:36.339653 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:05:36.343859 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.340349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.27.110.254:42957)
0812 07:05:36.340489 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.340493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.340715 (+ 222us) server_negotiation.cc:407] Connection header received
0812 07:05:36.340869 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.340872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.340961 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.341054 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:05:36.341340 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.341856 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.342723 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.342955 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.343051 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.343450 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:05:36.343569 (+ 119us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 07:05:36.343774 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56136: token expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 07:05:36.343953 24664 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56136: token expired
W20260812 07:05:36.344357 23709 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.110.254:42957)
I20260812 07:05:36.352108 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.344693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.27.110.254:42957)
0812 07:05:36.344826 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.344830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.344884 (+ 54us) server_negotiation.cc:407] Connection header received
0812 07:05:36.344971 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.344973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.345008 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.345072 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:36.345680 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.346193 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.346980 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.347265 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.347346 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.348648 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:36.348652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:36.348655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:36.348686 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:36.350333 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.350823 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.350827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.350828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.350915 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:36.351254 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:36.351258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:36.351260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:36.351435 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:36.351575 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.351826 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.351940 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:05:36.354564 23849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:05:36.358179 23849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:05:36.366640 24082 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb (DEFAULT_TABLE table=test-table [id=aee9fcc459be417cae53aa42db3bc1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:36.367364 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:36.370375 23998 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Bootstrap starting.
I20260812 07:05:36.370788 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.365507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38629 (local address 127.0.0.1:38150)
0812 07:05:36.365640 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.365657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:36.365743 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:36.366815 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:36.366818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:36.366839 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:36.366979 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.366986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.368235 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.368241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.369127 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.369136 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.370294 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.370298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.370421 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.370496 (+ 75us) client_negotiation.cc:769] Sending connection context
0812 07:05:36.370551 (+ 55us) client_negotiation.cc:240] Negotiation successful
0812 07:05:36.370656 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 07:05:36.371399 23998 tablet_bootstrap.cc:654] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:36.371484 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.365743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.27.110.195:38629)
0812 07:05:36.366371 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.366383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.366410 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:05:36.366487 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.366491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.366668 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.366897 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:36.367067 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.368123 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.369259 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.370146 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.370997 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.371183 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.371244 (+ 61us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.371347 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":188,"threads_started":1}
I20260812 07:05:36.372094 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.365751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42085 (local address 127.0.0.1:58324)
0812 07:05:36.366128 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.366142 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:36.366236 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:36.367105 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:36.367108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:36.367127 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:36.367229 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.367238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.369103 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.369107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.369872 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.369881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.371703 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.371707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.371798 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.371829 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 07:05:36.371894 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 07:05:36.371964 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":113,"threads_started":1}
I20260812 07:05:36.372200 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.366092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.27.110.194:42085)
0812 07:05:36.366480 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.366492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.366514 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:05:36.366589 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.366594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.366784 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.367607 (+ 823us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:36.367787 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.368965 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.370000 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.371525 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.371715 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.371879 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.371932 (+ 53us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.372025 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260812 07:05:36.374711 24543 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb (DEFAULT_TABLE table=test-table [id=aee9fcc459be417cae53aa42db3bc1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:36.374912 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb (DEFAULT_TABLE table=test-table [id=aee9fcc459be417cae53aa42db3bc1ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:05:36.375626 23998 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: No bootstrap required, opened a new log
I20260812 07:05:36.375734 23998 ts_tablet_manager.cc:1397] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260812 07:05:36.375918 24543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:36.375985 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:05:36.376214 23998 raft_consensus.cc:348] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.376341 23998 raft_consensus.cc:374] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:36.376385 23998 raft_consensus.cc:729] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba63cfef11da43ac93362cc54d2f817a, State: Initialized, Role: FOLLOWER
I20260812 07:05:36.376526 23998 consensus_queue.cc:260] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.376783 23998 ts_tablet_manager.cc:1428] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260812 07:05:36.383222 24680 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Bootstrap starting.
I20260812 07:05:36.384249 24681 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Bootstrap starting.
I20260812 07:05:36.385952 24680 tablet_bootstrap.cc:654] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:36.386381 24681 tablet_bootstrap.cc:654] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Neither blocks nor log segments found. Creating new log.
I20260812 07:05:36.386775 24680 log.cc:824] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:36.387174 24681 log.cc:824] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:36.388803 24680 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: No bootstrap required, opened a new log
I20260812 07:05:36.388967 24681 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: No bootstrap required, opened a new log
I20260812 07:05:36.389003 24680 ts_tablet_manager.cc:1397] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260812 07:05:36.389194 24681 ts_tablet_manager.cc:1397] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 07:05:36.392221 24680 raft_consensus.cc:348] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.392220 24681 raft_consensus.cc:348] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.392443 24680 raft_consensus.cc:374] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:36.392470 24681 raft_consensus.cc:374] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:05:36.392478 24680 raft_consensus.cc:729] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8cb6174519c4084a53039c198d3570e, State: Initialized, Role: FOLLOWER
I20260812 07:05:36.392545 24681 raft_consensus.cc:729] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7d31bddb3d48899dac1796f03c71a3, State: Initialized, Role: FOLLOWER
I20260812 07:05:36.392961 24680 consensus_queue.cc:260] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.392972 24681 consensus_queue.cc:260] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.393647 24427 heartbeater.cc:503] Master 127.27.110.254:42957 was elected leader, sending a full tablet report...
I20260812 07:05:36.393949 24681 ts_tablet_manager.cc:1428] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:05:36.393966 24680 ts_tablet_manager.cc:1428] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260812 07:05:36.394623 24657 heartbeater.cc:503] Master 127.27.110.254:42957 was elected leader, sending a full tablet report...
W20260812 07:05:36.397867 24183 tablet.cc:2369] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:05:36.528640 24687 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:36.528885 24687 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.530146 24687 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085), ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599)
I20260812 07:05:36.534106 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.530642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:60965 (local address 127.27.110.193:39599)
0812 07:05:36.530936 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.530939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.530950 (+ 11us) server_negotiation.cc:407] Connection header received
0812 07:05:36.531000 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.531002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.531054 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.531226 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:36.531515 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.532039 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.533076 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.533526 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.533664 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.533872 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.533910 (+ 38us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.533987 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":133,"threads_started":1}
I20260812 07:05:36.534097 24662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.530465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39599 (local address 127.27.110.195:60965)
0812 07:05:36.530636 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.530650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:36.530734 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:36.531237 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:36.531240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:36.531257 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:36.531386 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.531392 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.532159 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.532163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.532931 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.532940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.533656 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.533660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.533766 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.533788 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:05:36.533859 (+ 71us) client_negotiation.cc:240] Negotiation successful
0812 07:05:36.533941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 07:05:36.534427 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.530567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:48695 (local address 127.27.110.194:42085)
0812 07:05:36.530710 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.530716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.531313 (+ 597us) server_negotiation.cc:407] Connection header received
0812 07:05:36.531441 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.531450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.531497 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.531586 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:05:36.531864 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.532350 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.533386 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.533811 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.533944 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.533999 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.534190 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.534300 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:05:36.534521 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.530465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42085 (local address 127.27.110.195:48695)
0812 07:05:36.531199 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.531211 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:36.531344 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:36.531596 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:36.531599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:36.531616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:05:36.531746 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.531751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.532470 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.532474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.533252 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:36.533261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.533937 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.533941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:36.534044 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.534066 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:05:36.534308 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 07:05:36.534377 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":365,"thread_start_us":124,"threads_started":1}
I20260812 07:05:36.534838 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd3ba5f0b4c411db3c8015baa38c0bb" candidate_uuid: "b8cb6174519c4084a53039c198d3570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba63cfef11da43ac93362cc54d2f817a" is_pre_election: true
I20260812 07:05:36.535267 24107 raft_consensus.cc:2454] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8cb6174519c4084a53039c198d3570e in term 0.
I20260812 07:05:36.535305 24339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd3ba5f0b4c411db3c8015baa38c0bb" candidate_uuid: "b8cb6174519c4084a53039c198d3570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7d31bddb3d48899dac1796f03c71a3" is_pre_election: true
I20260812 07:05:36.535610 24339 raft_consensus.cc:2454] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8cb6174519c4084a53039c198d3570e in term 0.
I20260812 07:05:36.535845 24467 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cb6174519c4084a53039c198d3570e, ba63cfef11da43ac93362cc54d2f817a; no voters:
I20260812 07:05:36.536188 24687 raft_consensus.cc:2781] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:05:36.536278 24687 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:05:36.536321 24687 raft_consensus.cc:3037] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:36.537401 24687 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.537758 24687 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [CANDIDATE]: Term 1 election: Requested vote from peers bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085), ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599)
I20260812 07:05:36.538143 24107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd3ba5f0b4c411db3c8015baa38c0bb" candidate_uuid: "b8cb6174519c4084a53039c198d3570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba63cfef11da43ac93362cc54d2f817a"
I20260812 07:05:36.538251 24107 raft_consensus.cc:3037] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:36.539237 24107 raft_consensus.cc:2454] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8cb6174519c4084a53039c198d3570e in term 1.
I20260812 07:05:36.539489 24339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd3ba5f0b4c411db3c8015baa38c0bb" candidate_uuid: "b8cb6174519c4084a53039c198d3570e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7d31bddb3d48899dac1796f03c71a3"
I20260812 07:05:36.539515 24467 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8cb6174519c4084a53039c198d3570e, ba63cfef11da43ac93362cc54d2f817a; no voters:
I20260812 07:05:36.539613 24339 raft_consensus.cc:3037] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:05:36.539717 24687 raft_consensus.cc:2781] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:05:36.539964 24687 raft_consensus.cc:686] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 1 LEADER]: Becoming Leader. State: Replica: b8cb6174519c4084a53039c198d3570e, State: Running, Role: LEADER
I20260812 07:05:36.540328 24687 consensus_queue.cc:237] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:36.540796 24339 raft_consensus.cc:2454] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8cb6174519c4084a53039c198d3570e in term 1.
I20260812 07:05:36.543851 23849 catalog_manager.cc:5495] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e reported cstate change: term changed from 0 to 1, leader changed from <none> to b8cb6174519c4084a53039c198d3570e (127.27.110.195). New cstate: current_term: 1 leader_uuid: "b8cb6174519c4084a53039c198d3570e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } health_report { overall_health: UNKNOWN } } }
I20260812 07:05:36.563480 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.559757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.27.110.195:38629)
0812 07:05:36.559916 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:36.559920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:36.559958 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:05:36.560034 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:36.560036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:36.560109 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:36.560207 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:05:36.560451 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.561045 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.562003 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:36.562227 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:36.562309 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:36.562596 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:05:36.563007 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:05:36.563174 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:36.563226 (+ 52us) server_negotiation.cc:299] Negotiation successful
0812 07:05:36.563279 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 07:05:36.568674 24661 tablet.cc:2369] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:05:36.569695 24107 raft_consensus.cc:1261] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Refusing update from remote peer b8cb6174519c4084a53039c198d3570e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:05:36.569700 24339 raft_consensus.cc:1261] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Refusing update from remote peer b8cb6174519c4084a53039c198d3570e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:05:36.570418 24687 consensus_queue.cc:1035] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:05:36.570621 24690 consensus_queue.cc:1035] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:05:36.588068 23849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:05:36.588266 23849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:05:36.590677 23849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Sending DeleteTablet for 3 replicas of tablet 9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:05:36.591476 24543 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:36 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:38150
I20260812 07:05:36.591495 24296 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:36 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:58324
I20260812 07:05:36.591621 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:05:36 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:52562
I20260812 07:05:36.592002 24710 tablet_replica.cc:330] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: stopping tablet replica
I20260812 07:05:36.592270 24716 tablet_replica.cc:330] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: stopping tablet replica
I20260812 07:05:36.592288 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 23913
I20260812 07:05:36.592399 24716 raft_consensus.cc:2229] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:36.592519 24716 raft_consensus.cc:2258] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:36.592329 24710 raft_consensus.cc:2229] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:05:36.598150 24710 raft_consensus.cc:2258] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:36.598706 24714 tablet_replica.cc:330] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: stopping tablet replica
I20260812 07:05:36.599186 24714 raft_consensus.cc:2229] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:05:36.599591 24714 raft_consensus.cc:2258] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:05:36.616117 24714 ts_tablet_manager.cc:1905] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:36.617319 24710 ts_tablet_manager.cc:1905] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:05:36.619171 24714 ts_tablet_manager.cc:1918] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:36.619298 24714 log.cc:1196] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:05:36.619655 24714 ts_tablet_manager.cc:1939] T 9cd3ba5f0b4c411db3c8015baa38c0bb P bb7d31bddb3d48899dac1796f03c71a3: Deleting consensus metadata
I20260812 07:05:36.620307 24710 ts_tablet_manager.cc:1918] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:05:36.620427 24710 log.cc:1196] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:05:36.620816 24710 ts_tablet_manager.cc:1939] T 9cd3ba5f0b4c411db3c8015baa38c0bb P b8cb6174519c4084a53039c198d3570e: Deleting consensus metadata
I20260812 07:05:36.621176 23821 catalog_manager.cc:4841] TS bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): tablet 9cd3ba5f0b4c411db3c8015baa38c0bb (table test-table [id=aee9fcc459be417cae53aa42db3bc1ca]) successfully deleted
I20260812 07:05:36.622167 23825 catalog_manager.cc:4841] TS b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): tablet 9cd3ba5f0b4c411db3c8015baa38c0bb (table test-table [id=aee9fcc459be417cae53aa42db3bc1ca]) successfully deleted
W20260812 07:05:36.623021 23823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:39599 (error 108)
I20260812 07:05:36.623274 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 24193
I20260812 07:05:36.635738 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:36.635304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39599 (local address 127.0.0.1:52576)
0812 07:05:36.635541 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:36.635627 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:39599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260812 07:05:36.635959 23823 catalog_manager.cc:4584] TS ba63cfef11da43ac93362cc54d2f817a (127.27.110.193:39599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cd3ba5f0b4c411db3c8015baa38c0bb: Network error: Client connection negotiation failed: client connection to 127.27.110.193:39599: connect: Connection refused (error 111)
I20260812 07:05:36.642974 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 24433
I20260812 07:05:36.657158 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 23784
I20260812 07:05:56.672281 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42957
--webserver_interface=127.27.110.254
--webserver_port=36065
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:05:56.795370 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:56.795682 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:56.795727 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:56.799868 24722 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 07:05:56.799991 24722 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 07:05:56.800021 24722 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 07:05:56.800050 24722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:05:56.800109 24722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:56.800137 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:56.800165 24722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:05:56.800212 24722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:05:56.805769 24722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=36065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:56.807029 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:56.808173 24722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:05:56.818210 24722 server_base.cc:1034] running on GCE node
W20260812 07:05:56.818154 24729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:56.818163 24730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:56.818145 24734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:56.819233 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:56.820429 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:56.821651 24722 hybrid_clock.cc:648] HybridClock initialized: now 1786518356821637 us; error 28 us; skew 500 ppm
Aug 12 07:05:56 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518356, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:56.829324 24722 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:05:56.830317 24722 webserver.cc:460] Webserver started at http://127.27.110.254:36065/ using document root <none> and password file <none>
I20260812 07:05:56.830956 24722 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:56.831064 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:56.835289 24722 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260812 07:05:56.837633 24747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:56.838999 24722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:05:56.839205 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "572fb8ecc78a49e2b08fd6c24f9439d1"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "ef9e2f0be3df293c8815396cf4fe0043"
server_key_iv: "3afd6c2b1c963bae77d4c5674a6ff9ce"
server_key_version: "encryptionkey@0"
I20260812 07:05:56.839800 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:56.871150 24722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:56.874801 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:56.875022 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:56.882735 24722 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42957
I20260812 07:05:56.883036 24832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42957 every 8 connection(s)
I20260812 07:05:56.883997 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:05:56.888447 24833 sys_catalog.cc:263] Verifying existing consensus state
I20260812 07:05:56.890146 24833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap starting.
I20260812 07:05:56.891719 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 24722
I20260812 07:05:56.892221 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:39599
--local_ip_for_outbound_sockets=127.27.110.193
--tserver_master_addrs=127.27.110.254:42957
--webserver_port=34541
--webserver_interface=127.27.110.193
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:05:56.915777 24833 log.cc:824] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:56.930773 24833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:05:56.931303 24833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Bootstrap complete.
I20260812 07:05:56.936952 24833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:56.937487 24833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572fb8ecc78a49e2b08fd6c24f9439d1, State: Initialized, Role: FOLLOWER
I20260812 07:05:56.938100 24833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:56.938292 24833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:05:56.938385 24833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:05:56.938493 24833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 2 FOLLOWER]: Advancing to term 3
I20260812 07:05:56.941220 24833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:56.941599 24833 leader_election.cc:302] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 572fb8ecc78a49e2b08fd6c24f9439d1; no voters:
I20260812 07:05:56.941980 24833 leader_election.cc:288] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 07:05:56.942152 24842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 07:05:56.942443 24842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [term 3 LEADER]: Becoming Leader. State: Replica: 572fb8ecc78a49e2b08fd6c24f9439d1, State: Running, Role: LEADER
I20260812 07:05:56.942772 24842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } }
I20260812 07:05:56.943413 24833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:05:56.944350 24844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } } }
I20260812 07:05:56.944480 24844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:56.944942 24846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 572fb8ecc78a49e2b08fd6c24f9439d1. Latest consensus state: current_term: 3 leader_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "572fb8ecc78a49e2b08fd6c24f9439d1" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42957 } } }
I20260812 07:05:56.945055 24846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1 [sys.catalog]: This master's current role is: LEADER
I20260812 07:05:56.945314 24865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:05:56.949596 24865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:05:56.950330 24865 catalog_manager.cc:1259] Loaded cluster ID: 4f95bdc1595f4b8b81a2271d12462696
I20260812 07:05:56.950403 24865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:05:56.952579 24865 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:05:56.953203 24865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Loaded TSK: 0
I20260812 07:05:56.953295 24865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 572fb8ecc78a49e2b08fd6c24f9439d1: Loaded TSK: 1
F20260812 07:05:56.953914 24865 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3efbc119ad google::LogMessage::Fail() at ??:0
@ 0x7f3efbc1488a google::LogMessage::SendToLog() at ??:0
@ 0x7f3efbc114de google::LogMessage::Flush() at ??:0
@ 0x7f3efbc12a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3efe8d518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3efe8cdf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3f065767bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3f0655099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3f06551178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f3f06592df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f3f065b182b std::function<>::operator()() at ??:0
@ 0x7f3f06550b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3f06551f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3f0654e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3f06592640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3efde633d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3efde63c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3efde653f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3efde54512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3eff4986db start_thread at ??:0
@ 0x7f3efa70f61f clone at ??:0
W20260812 07:05:57.054260 24836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:57.054527 24836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:57.054575 24836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:57.059580 24836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:57.059638 24836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:57.059703 24836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:05:57.065913 24836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:39599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=34541
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:57.067240 24836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:57.068384 24836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:57.079272 24933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:57.079262 24932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:57.079262 24936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:57.079639 24836 server_base.cc:1034] running on GCE node
I20260812 07:05:57.080253 24836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:57.081055 24836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:57.082259 24836 hybrid_clock.cc:648] HybridClock initialized: now 1786518357082231 us; error 43 us; skew 500 ppm
Aug 12 07:05:57 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:57.085484 24836 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:05:57.086598 24836 webserver.cc:460] Webserver started at http://127.27.110.193:34541/ using document root <none> and password file <none>
I20260812 07:05:57.087335 24836 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:57.087411 24836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:57.091840 24836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260812 07:05:57.094218 24950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.095749 24836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:05:57.095886 24836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "ba63cfef11da43ac93362cc54d2f817a"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "a4e909c46e8ed9cf8c30a597e441f669"
server_key_iv: "2fbdc7e54944ac81114b12966932bd2a"
server_key_version: "encryptionkey@0"
I20260812 07:05:57.096475 24836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:57.108687 24836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:57.112200 24836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:57.112421 24836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:57.113114 24836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:57.114300 24960 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260812 07:05:57.118546 24836 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 07:05:57.118619 24836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260812 07:05:57.118669 24836 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 07:05:57.120419 24836 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 07:05:57.120478 24836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260812 07:05:57.120731 24960 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Bootstrap starting.
I20260812 07:05:57.131875 24836 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:39599
I20260812 07:05:57.132035 25166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:39599 every 8 connection(s)
I20260812 07:05:57.133167 24836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:05:57.142102 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 24836
I20260812 07:05:57.142660 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:42085
--local_ip_for_outbound_sockets=127.27.110.194
--tserver_master_addrs=127.27.110.254:42957
--webserver_port=39663
--webserver_interface=127.27.110.194
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:05:57.143795 24960 log.cc:824] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:05:57.152958 24960 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 07:05:57.153499 24960 tablet_bootstrap.cc:492] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Bootstrap complete.
I20260812 07:05:57.155203 24960 ts_tablet_manager.cc:1397] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.009s
I20260812 07:05:57.158564 24960 raft_consensus.cc:348] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:57.159183 24960 raft_consensus.cc:729] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba63cfef11da43ac93362cc54d2f817a, State: Initialized, Role: FOLLOWER
I20260812 07:05:57.159751 24960 consensus_queue.cc:260] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:57.160872 24960 ts_tablet_manager.cc:1428] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260812 07:05:57.205857 25171 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:05:57.135555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:35035)
0812 07:05:57.136169 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.136229 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:57.137305 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:57.204868 (+ 67563us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":166,"threads_started":1}
W20260812 07:05:57.206462 25168 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:42957 (0 consecutive failures): Network error: Failed to ping master at 127.27.110.254:42957: Client connection negotiation failed: client connection to 127.27.110.254:42957: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 07:05:57.207345 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.207012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:58663)
0812 07:05:57.207193 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.207275 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 07:05:57.208056 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.207830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:36445)
0812 07:05:57.207945 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.208000 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 07:05:57.208324 25168 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:05:57.261411 25174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:57.261670 25174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:57.261713 25174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:57.265798 25174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:57.265862 25174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:57.265946 25174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:05:57.271114 25174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:42085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=39663
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:57.272132 25174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:57.273156 25174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:57.282869 25192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:57.282908 25189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:57.282910 25187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:57.283245 25174 server_base.cc:1034] running on GCE node
I20260812 07:05:57.283747 25174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:57.284373 25174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:57.285607 25174 hybrid_clock.cc:648] HybridClock initialized: now 1786518357285566 us; error 35 us; skew 500 ppm
Aug 12 07:05:57 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:57.288416 25174 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:05:57.289273 25174 webserver.cc:460] Webserver started at http://127.27.110.194:39663/ using document root <none> and password file <none>
I20260812 07:05:57.289831 25174 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:57.289893 25174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:57.294006 25174 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:05:57.296348 25206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.297762 25174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 07:05:57.298007 25174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "bb7d31bddb3d48899dac1796f03c71a3"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "358483acb81df9e38d1cc3cc4ebd3898"
server_key_iv: "8ad4fc4589acf4f24577add87f8baa2d"
server_key_version: "encryptionkey@0"
I20260812 07:05:57.298552 25174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:57.314412 25174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:57.317430 25174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:57.317663 25174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:57.318241 25174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:57.319046 25174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:57.319139 25174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.319206 25174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:57.319254 25174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.328682 25174 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:42085
I20260812 07:05:57.328769 25404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:42085 every 8 connection(s)
I20260812 07:05:57.329689 25174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:05:57.331256 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 25174
I20260812 07:05:57.331737 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:38629
--local_ip_for_outbound_sockets=127.27.110.195
--tserver_master_addrs=127.27.110.254:42957
--webserver_port=42387
--webserver_interface=127.27.110.195
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:45907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:05:57.333508 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.332030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:52811)
0812 07:05:57.332733 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.332837 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":150,"threads_started":1}
W20260812 07:05:57.334100 25408 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:42957 (0 consecutive failures): Network error: Failed to ping master at 127.27.110.254:42957: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
I20260812 07:05:57.334755 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.334473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:52137)
0812 07:05:57.334642 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.334696 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260812 07:05:57.335480 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.335296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:37853)
0812 07:05:57.335390 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.335433 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260812 07:05:57.335695 25408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 07:05:57.448675 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:05:57.448949 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:05:57.449019 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:05:57.453246 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:05:57.453320 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:05:57.453406 25411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:05:57.458528 25411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:38629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=42387
--tserver_master_addrs=127.27.110.254:42957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:05:57.459656 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:05:57.460677 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:05:57.470672 25423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:57.470763 25411 server_base.cc:1034] running on GCE node
W20260812 07:05:57.470723 25425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:05:57.470674 25422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:05:57.471352 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:05:57.472014 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:05:57.473184 25411 hybrid_clock.cc:648] HybridClock initialized: now 1786518357473166 us; error 33 us; skew 500 ppm
Aug 12 07:05:57 dist-test-slave-w4v5 krb5kdc[22798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:05:57.476091 25411 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:05:57.476955 25411 webserver.cc:460] Webserver started at http://127.27.110.195:42387/ using document root <none> and password file <none>
I20260812 07:05:57.477478 25411 fs_manager.cc:359] Metadata directory not provided
I20260812 07:05:57.477540 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:05:57.481276 25411 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:05:57.483268 25435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.484513 25411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:05:57.484654 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "b8cb6174519c4084a53039c198d3570e"
format_stamp: "Formatted at 2026-08-12 07:05:14 on dist-test-slave-w4v5"
server_key: "63c9a3c5eee74820e37c2fd5e4441e08"
server_key_iv: "3efba96ff7d58c4e7838b6d2cab3c3bc"
server_key_version: "encryptionkey@0"
I20260812 07:05:57.485176 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:05:57.505445 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:05:57.508747 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:05:57.508955 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:05:57.509569 25411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:05:57.510422 25411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:05:57.510507 25411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.510579 25411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:05:57.510620 25411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:05:57.519973 25411 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:38629
I20260812 07:05:57.520004 25634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:38629 every 8 connection(s)
I20260812 07:05:57.521088 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:05:57.524700 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.523228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:42039)
0812 07:05:57.523880 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.523995 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":112,"threads_started":1}
W20260812 07:05:57.525174 25635 heartbeater.cc:644] Failed to heartbeat to 127.27.110.254:42957 (0 consecutive failures): Network error: Failed to ping master at 127.27.110.254:42957: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
I20260812 07:05:57.525839 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.525602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:39257)
0812 07:05:57.525733 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.525783 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 07:05:57.526482 25641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:57.526293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:39639)
0812 07:05:57.526389 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:57.526432 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 07:05:57.526660 25635 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 07:05:57.530308 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 25411
I20260812 07:05:58.209364 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.208932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:57457)
0812 07:05:58.209204 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:58.209260 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
I20260812 07:05:58.336649 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.336249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:50783)
0812 07:05:58.336499 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:58.336552 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260812 07:05:58.434603 25651 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:05:58.434839 25651 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:05:58.436056 25651 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629), bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
Aug 12 07:05:58 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.194@KRBTEST.COM
Aug 12 07:05:58 dist-test-slave-w4v5 krb5kdc[22798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518357, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.195@KRBTEST.COM
I20260812 07:05:58.449683 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.436284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42085 (local address 127.27.110.193:46765)
0812 07:05:58.436979 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:58.436992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:58.437089 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:58.438396 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:58.438402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:58.438960 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:58.439529 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:58.439545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.440932 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.440936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:58.441979 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:58.441991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.442692 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.442696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:58.442924 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:58.443586 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:58.443603 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:58.445122 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:58.447308 (+ 2186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:58.447314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:58.447326 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:58.447604 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:58.447961 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:58.447964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:58.447966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:58.448109 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:58.448729 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:58.448735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:58.448966 (+ 231us) client_negotiation.cc:769] Sending connection context
0812 07:05:58.449148 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 07:05:58.449468 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":555,"spinlock_wait_cycles":24064,"thread_start_us":56,"threads_started":1}
I20260812 07:05:58.449709 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.436493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:46765 (local address 127.27.110.194:42085)
0812 07:05:58.437013 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:58.437034 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:58.437069 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:05:58.437765 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:58.437771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:58.438072 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:58.438369 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:58.439708 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.440743 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.442106 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.442567 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.442744 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:58.445391 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:58.445396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:58.445408 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:58.445436 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:58.447188 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:58.447726 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:58.447732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:58.447737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:58.447857 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:58.448216 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:58.448219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:58.448221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:58.448610 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:58.448755 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:58.449276 (+ 521us) server_negotiation.cc:299] Negotiation successful
0812 07:05:58.449539 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":163,"threads_started":1}
I20260812 07:05:58.449832 25652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.436524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:44785 (local address 127.27.110.195:38629)
0812 07:05:58.436865 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0812 07:05:58.436881 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:05:58.436927 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:05:58.437671 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:05:58.437677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:05:58.438037 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:05:58.438316 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:05:58.439710 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.440773 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.442143 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.442591 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.442767 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:58.445647 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:05:58.445652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:05:58.445663 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:05:58.445691 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:05:58.447609 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:58.448073 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:58.448085 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:58.448089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:58.448200 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:05:58.448512 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:05:58.448515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:05:58.448517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:05:58.448880 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:05:58.449033 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 07:05:58.449368 (+ 335us) server_negotiation.cc:299] Negotiation successful
0812 07:05:58.449658 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":154,"threads_started":1}
I20260812 07:05:58.449751 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.436266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:38629 (local address 127.27.110.193:44785)
0812 07:05:58.436408 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:58.436423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:05:58.436582 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:05:58.438397 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:05:58.438403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:05:58.438960 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:05:58.439529 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:58.439544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.440985 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.440989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:58.441983 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:05:58.441994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:05:58.442731 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:05:58.442735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:05:58.442922 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:05:58.443735 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:05:58.443756 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:05:58.445436 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:05:58.447733 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:58.447738 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:58.447740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:58.447945 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:58.448311 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:05:58.448314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:05:58.448316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:05:58.448405 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:05:58.449003 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:05:58.449007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:05:58.449101 (+ 94us) client_negotiation.cc:769] Sending connection context
0812 07:05:58.449220 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 07:05:58.449568 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":259,"spinlock_wait_cycles":57088}
W20260812 07:05:58.452979 24956 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
W20260812 07:05:58.453641 24954 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:05:58.453718 24954 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba63cfef11da43ac93362cc54d2f817a; no voters: b8cb6174519c4084a53039c198d3570e, bb7d31bddb3d48899dac1796f03c71a3
I20260812 07:05:58.453949 25651 raft_consensus.cc:2726] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:05:58.527676 25658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:58.527280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:45795)
0812 07:05:58.527529 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:58.527579 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":83,"threads_started":1}
I20260812 07:05:59.210587 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:59.210106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:43943)
0812 07:05:59.210384 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:59.210485 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260812 07:05:59.337774 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:59.337374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:40177)
0812 07:05:59.337619 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:59.337677 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260812 07:05:59.528769 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:05:59.528346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:59487)
0812 07:05:59.528608 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 07:05:59.528664 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260812 07:06:00.061143 25662 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:00.061334 25662 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:06:00.061775 25662 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629), bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
W20260812 07:06:00.062743 24956 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
W20260812 07:06:00.062844 24954 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:06:00.062913 24954 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba63cfef11da43ac93362cc54d2f817a; no voters: b8cb6174519c4084a53039c198d3570e, bb7d31bddb3d48899dac1796f03c71a3
I20260812 07:06:00.063158 25662 raft_consensus.cc:2726] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:06:00.211867 25664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:00.211467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:35771)
0812 07:06:00.211717 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:00.211770 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":83,"threads_started":1}
I20260812 07:06:00.339061 25665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:00.338525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:47761)
0812 07:06:00.338892 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:00.338947 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":127,"threads_started":1}
I20260812 07:06:00.529829 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:00.529443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:54277)
0812 07:06:00.529688 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:00.529741 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260812 07:06:01.213205 25670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:01.212732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:39143)
0812 07:06:01.213018 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:01.213101 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260812 07:06:01.340404 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:01.339953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:55409)
0812 07:06:01.340238 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:01.340304 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260812 07:06:01.531402 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:01.530724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:43159)
0812 07:06:01.531042 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:01.531295 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260812 07:06:02.214589 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:02.214130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:44395)
0812 07:06:02.214438 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:02.214491 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260812 07:06:02.341614 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:02.341211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:32835)
0812 07:06:02.341471 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:02.341521 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
I20260812 07:06:02.532611 25681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:02.532176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:60743)
0812 07:06:02.532424 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:02.532515 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260812 07:06:03.215942 25682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:03.215513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:43967)
0812 07:06:03.215778 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:03.215837 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260812 07:06:03.234615 25685 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:03.234782 25685 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:06:03.235301 25685 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629), bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
W20260812 07:06:03.236073 24954 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
W20260812 07:06:03.236194 24956 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:06:03.236307 24956 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba63cfef11da43ac93362cc54d2f817a; no voters: b8cb6174519c4084a53039c198d3570e, bb7d31bddb3d48899dac1796f03c71a3
I20260812 07:06:03.236465 25685 raft_consensus.cc:2726] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:06:03.342824 25687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:03.342353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:41711)
0812 07:06:03.342630 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:03.342729 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":80,"threads_started":1}
I20260812 07:06:03.533779 25688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:03.533354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:39365)
0812 07:06:03.533608 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:03.533684 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
I20260812 07:06:04.217120 25689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:04.216704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:41829)
0812 07:06:04.216967 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:04.217024 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260812 07:06:04.344074 25694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:04.343612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:37101)
0812 07:06:04.343916 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:04.343971 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":77,"threads_started":1}
I20260812 07:06:04.535008 25695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:04.534540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:33911)
0812 07:06:04.534855 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:04.534911 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
I20260812 07:06:04.973349 25700 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:04.973507 25700 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:06:04.974010 25700 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629), bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
W20260812 07:06:04.974968 24956 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
W20260812 07:06:04.975116 24954 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:06:04.975193 24954 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba63cfef11da43ac93362cc54d2f817a; no voters: b8cb6174519c4084a53039c198d3570e, bb7d31bddb3d48899dac1796f03c71a3
I20260812 07:06:04.975342 25700 raft_consensus.cc:2726] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:06:05.218386 25701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.217964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:37961)
0812 07:06:05.218226 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.218285 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":93,"threads_started":1}
I20260812 07:06:05.345348 25704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.344857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:54273)
0812 07:06:05.345197 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.345256 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
I20260812 07:06:05.536290 25705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:05.535851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:55707)
0812 07:06:05.536127 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:05.536192 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260812 07:06:06.219802 25706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:06.219295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:52157)
0812 07:06:06.219628 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:06.219701 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
I20260812 07:06:06.346714 25709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:06.346258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:34415)
0812 07:06:06.346547 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:06.346606 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
I20260812 07:06:06.537479 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:06.537082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:45055)
0812 07:06:06.537330 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:06.537383 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260812 07:06:07.221055 25712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:07.220615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:38239)
0812 07:06:07.220898 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:07.220957 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260812 07:06:07.347911 25713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:07.347510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:39337)
0812 07:06:07.347761 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:07.347817 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":77,"threads_started":1}
I20260812 07:06:07.538784 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:07.538317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:37265)
0812 07:06:07.538628 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:07.538689 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260812 07:06:08.222244 25717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:08.221810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:47437)
0812 07:06:08.222087 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:08.222144 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":78,"threads_started":1}
I20260812 07:06:08.349128 25720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:08.348672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:54149)
0812 07:06:08.348963 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:08.349023 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":99,"threads_started":1}
I20260812 07:06:08.540086 25722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:08.539629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:59869)
0812 07:06:08.539914 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:08.539973 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260812 07:06:09.223634 25723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:09.223204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:42675)
0812 07:06:09.223476 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:09.223536 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
I20260812 07:06:09.350535 25726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:09.350019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:48581)
0812 07:06:09.350312 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:09.350376 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":100,"threads_started":1}
I20260812 07:06:09.541370 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:09.540915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:39689)
0812 07:06:09.541212 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:09.541270 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260812 07:06:09.541477 25729 raft_consensus.cc:482] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:09.541683 25729 raft_consensus.cc:504] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b8cb6174519c4084a53039c198d3570e" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 38629 } } peers { permanent_uuid: "bb7d31bddb3d48899dac1796f03c71a3" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42085 } } peers { permanent_uuid: "ba63cfef11da43ac93362cc54d2f817a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39599 } }
I20260812 07:06:09.542156 25729 leader_election.cc:288] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629), bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085)
W20260812 07:06:09.542902 24956 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b8cb6174519c4084a53039c198d3570e (127.27.110.195:38629): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
W20260812 07:06:09.543008 24954 leader_election.cc:341] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bb7d31bddb3d48899dac1796f03c71a3 (127.27.110.194:42085): Not found: Tablet not found: 9cd3ba5f0b4c411db3c8015baa38c0bb
I20260812 07:06:09.543150 24954 leader_election.cc:302] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ba63cfef11da43ac93362cc54d2f817a; no voters: b8cb6174519c4084a53039c198d3570e, bb7d31bddb3d48899dac1796f03c71a3
I20260812 07:06:09.543327 25729 raft_consensus.cc:2726] T 9cd3ba5f0b4c411db3c8015baa38c0bb P ba63cfef11da43ac93362cc54d2f817a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 07:06:10.225104 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:10.224545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:44671)
0812 07:06:10.224936 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:10.225001 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":112,"threads_started":1}
I20260812 07:06:10.351850 25735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:10.351400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:51879)
0812 07:06:10.351697 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:10.351752 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":116,"threads_started":1}
I20260812 07:06:10.542577 25736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:10.542158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:57053)
0812 07:06:10.542411 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:10.542470 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260812 07:06:11.226372 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:11.225920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:57241)
0812 07:06:11.226211 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:11.226270 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260812 07:06:11.353240 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:11.352749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:42733)
0812 07:06:11.353058 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:11.353125 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260812 07:06:11.543887 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:11.543437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:50639)
0812 07:06:11.543731 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:11.543787 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
I20260812 07:06:12.228010 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:12.227548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.193:41989)
0812 07:06:12.227832 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:12.227890 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260812 07:06:12.354574 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:12.354165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.194:50657)
0812 07:06:12.354415 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:12.354475 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260812 07:06:12.541208 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 24836
I20260812 07:06:12.547145 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:12.544886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:42957 (local address 127.27.110.195:37431)
0812 07:06:12.546859 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:12.546951 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.254:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1865,"thread_start_us":1566,"threads_started":1}
I20260812 07:06:12.557866 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 25174
I20260812 07:06:12.571875 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 25411
2026-08-12T07:06:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59945 ms)
[----------] 30 tests from SecurityITest (365797 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:12 dist-test-slave-w4v5 krb5kdc[25752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:12 dist-test-slave-w4v5 krb5kdc[25752](info): set up 2 sockets
Aug 12 07:06:12 dist-test-slave-w4v5 krb5kdc[25752](info): commencing operation
krb5kdc: starting...
W20260812 07:06:13.970094 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:14.098340 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36341
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:14.210243 25779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:14.210554 25779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:14.210637 25779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:14.214625 25779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:14.214718 25779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:14.214757 25779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:14.214815 25779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:14.214852 25779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:14.220005 25779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:14.221036 25779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:14.222066 25779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:14.231901 25793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.231894 25788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.231875 25789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:14.231935 25779 server_base.cc:1034] running on GCE node
I20260812 07:06:14.232887 25779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:14.234047 25779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:14.235235 25779 hybrid_clock.cc:648] HybridClock initialized: now 1786518374235206 us; error 38 us; skew 500 ppm
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:14.238336 25779 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:14.239429 25779 webserver.cc:460] Webserver started at http://127.27.110.254:39019/ using document root <none> and password file <none>
I20260812 07:06:14.240020 25779 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:14.240083 25779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:14.240311 25779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:14.242449 25779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "52de66f2c043efcc1c144bd78b394494"
server_key_iv: "5b6d4b332b342db70c5e7bd89194f51f"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.243026 25779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "52de66f2c043efcc1c144bd78b394494"
server_key_iv: "5b6d4b332b342db70c5e7bd89194f51f"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.247298 25779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:06:14.249817 25805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.251287 25779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 07:06:14.251436 25779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "52de66f2c043efcc1c144bd78b394494"
server_key_iv: "5b6d4b332b342db70c5e7bd89194f51f"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.251554 25779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:14.270063 25779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:14.273124 25779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:14.273337 25779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:14.281041 25779 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36341
I20260812 07:06:14.281069 25896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36341 every 8 connection(s)
I20260812 07:06:14.282071 25779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:14.284970 25898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:14.285120 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 25779
I20260812 07:06:14.285274 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:14.285584 28091 external_mini_cluster.cc:1428] Setting key 78f44cd8ea69c5e6363e61fda1136ebe
I20260812 07:06:14.292466 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc: Bootstrap starting.
I20260812 07:06:14.295017 25898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc: Neither blocks nor log segments found. Creating new log.
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:14.296028 25898 log.cc:824] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:14.298352 25898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc: No bootstrap required, opened a new log
I20260812 07:06:14.301258 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:14.287174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.27.110.254:36341)
0812 07:06:14.287637 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:14.287656 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:14.287701 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:06:14.288515 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:14.288542 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:14.288999 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:14.289382 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:14.290100 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.291687 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.292635 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.293137 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.293317 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:14.296305 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:14.296312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:14.296331 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:14.296387 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:14.298244 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:14.298827 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:14.298833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:14.298846 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:14.298959 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:14.299298 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:14.299302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:14.299304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:14.299729 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:14.299942 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:14.300206 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 07:06:14.300510 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":138,"threads_started":1}
I20260812 07:06:14.302139 25898 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36341 } }
I20260812 07:06:14.302385 25898 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:14.302465 25898 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce4366a01c5f4fe4a0d5c1f818342bdc, State: Initialized, Role: FOLLOWER
I20260812 07:06:14.302894 25898 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36341 } }
I20260812 07:06:14.303103 25898 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:14.303200 25898 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:14.303299 25898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:14.304255 25898 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36341 } }
I20260812 07:06:14.304590 25898 leader_election.cc:302] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce4366a01c5f4fe4a0d5c1f818342bdc; no voters:
I20260812 07:06:14.304912 25898 leader_election.cc:288] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:14.305087 25907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:14.305341 25907 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [term 1 LEADER]: Becoming Leader. State: Replica: ce4366a01c5f4fe4a0d5c1f818342bdc, State: Running, Role: LEADER
I20260812 07:06:14.305668 25907 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36341 } }
I20260812 07:06:14.306115 25898 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:14.307493 25908 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36341 } } }
I20260812 07:06:14.307492 25912 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce4366a01c5f4fe4a0d5c1f818342bdc. Latest consensus state: current_term: 1 leader_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce4366a01c5f4fe4a0d5c1f818342bdc" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36341 } } }
I20260812 07:06:14.307644 25908 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:14.307644 25912 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:14.308128 25927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:14.310845 25927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:14.316963 25927 catalog_manager.cc:1347] Generated new cluster ID: ece34db1cf33465e93bc22e7a0accffe
I20260812 07:06:14.317044 25927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:14.336088 25927 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:14.338161 25927 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:14.352726 25927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc: Generated new TSK 0
I20260812 07:06:14.353598 25927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:06:14.396486 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36341
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:14.513207 25947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:14.513492 25947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:14.513586 25947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:14.517675 25947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:14.517763 25947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:14.517855 25947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:06:14.523063 25947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:14.524130 25947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:14.525141 25947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:14.534956 25962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.534956 25957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.534981 25958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:14.535447 25947 server_base.cc:1034] running on GCE node
I20260812 07:06:14.535831 25947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:14.536481 25947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:14.537679 25947 hybrid_clock.cc:648] HybridClock initialized: now 1786518374537667 us; error 28 us; skew 500 ppm
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:14.540485 25947 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:06:14.541450 25947 webserver.cc:460] Webserver started at http://127.27.110.193:40905/ using document root <none> and password file <none>
I20260812 07:06:14.542016 25947 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:14.542109 25947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:14.542351 25947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:14.544466 25947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "270d8fbc57964fc1ae345f94c13d758a"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "8b512bb722bff3d864966642076dfaa9"
server_key_iv: "3809b7ee4e62594243bf496f0c194554"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.545023 25947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "270d8fbc57964fc1ae345f94c13d758a"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "8b512bb722bff3d864966642076dfaa9"
server_key_iv: "3809b7ee4e62594243bf496f0c194554"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.548767 25947 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:06:14.551386 25974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.552681 25947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:06:14.552851 25947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "270d8fbc57964fc1ae345f94c13d758a"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "8b512bb722bff3d864966642076dfaa9"
server_key_iv: "3809b7ee4e62594243bf496f0c194554"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.552978 25947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:14.570554 25947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:14.573555 25947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:14.573778 25947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:14.574486 25947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:14.575392 25947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:14.575462 25947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.575534 25947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:14.575582 25947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.585003 25947 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:39183
I20260812 07:06:14.585036 26176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:39183 every 8 connection(s)
I20260812 07:06:14.585917 25947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:14.594698 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 25947
I20260812 07:06:14.594862 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:14.595203 28091 external_mini_cluster.cc:1428] Setting key a17b019d0895d9f24ebc4c682d47d083
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:14.600698 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:14.587914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:38715 (local address 127.27.110.254:36341)
0812 07:06:14.588081 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:14.588085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:14.588894 (+ 809us) server_negotiation.cc:407] Connection header received
0812 07:06:14.590183 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:14.590186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:14.590234 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:14.590308 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:14.591588 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.592125 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.593319 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.593554 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.593634 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:14.596584 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:14.596588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:14.596591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:14.596617 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:14.598140 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:14.598805 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:14.598808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:14.598809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:14.598914 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:14.599436 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:14.599442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:14.599445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:14.599732 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:14.599843 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:14.600355 (+ 512us) server_negotiation.cc:299] Negotiation successful
0812 07:06:14.600509 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:06:14.601706 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:14.588281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36341 (local address 127.27.110.193:38715)
0812 07:06:14.588728 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:14.588773 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:14.589971 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:14.590492 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:14.590499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:14.590861 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:14.591443 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:14.591457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.592276 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.592279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:14.593205 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:14.593216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.593722 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.593726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:14.593878 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:14.594531 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:14.594584 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:14.596408 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:14.598265 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:14.598275 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:14.598294 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:14.598697 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:14.599161 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:14.599167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:14.599169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:14.599333 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:14.599843 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:14.599848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:14.600091 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 07:06:14.600249 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 07:06:14.600512 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":114,"threads_started":1}
I20260812 07:06:14.603107 26177 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36341
I20260812 07:06:14.603377 26177 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:14.604033 26177 heartbeater.cc:511] Master 127.27.110.254:36341 requested a full tablet report, sending...
I20260812 07:06:14.605845 25834 ts_manager.cc:194] Registered new tserver with Master: 270d8fbc57964fc1ae345f94c13d758a (127.27.110.193:39183)
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:06:14.607573 25834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:38715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:06:14.625617 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36341
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:14.740859 26187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:14.741153 26187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:14.741200 26187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:14.745417 26187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:14.745492 26187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:14.745555 26187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:06:14.751514 26187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:14.752625 26187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:14.753602 26187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:14.764868 26197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.764873 26196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.765370 26199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:14.765972 26187 server_base.cc:1034] running on GCE node
I20260812 07:06:14.766417 26187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:14.767112 26187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:14.768303 26187 hybrid_clock.cc:648] HybridClock initialized: now 1786518374768286 us; error 25 us; skew 500 ppm
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:14.771188 26187 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:06:14.772161 26187 webserver.cc:460] Webserver started at http://127.27.110.194:38995/ using document root <none> and password file <none>
I20260812 07:06:14.772759 26187 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:14.772854 26187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:14.773092 26187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:14.775205 26187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "ee4d4324fc0b43b997c31c7ac64941f1"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "f49c7833d5ac33c57890d5350d1c938b"
server_key_iv: "dc5ff1255004ca9c41cc0d6ea0c5bac6"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.775787 26187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "ee4d4324fc0b43b997c31c7ac64941f1"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "f49c7833d5ac33c57890d5350d1c938b"
server_key_iv: "dc5ff1255004ca9c41cc0d6ea0c5bac6"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.779968 26187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:14.782593 26207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.784135 26187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:14.784286 26187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "ee4d4324fc0b43b997c31c7ac64941f1"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "f49c7833d5ac33c57890d5350d1c938b"
server_key_iv: "dc5ff1255004ca9c41cc0d6ea0c5bac6"
server_key_version: "encryptionkey@0"
I20260812 07:06:14.784456 26187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:14.806661 26187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:14.810405 26187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:14.810657 26187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:14.811373 26187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:14.812281 26187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:14.812330 26187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.812404 26187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:14.812453 26187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:14.821501 26187 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:44021
I20260812 07:06:14.821514 26415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:44021 every 8 connection(s)
I20260812 07:06:14.822482 26187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:14.822786 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 26187
I20260812 07:06:14.822921 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:14.823289 28091 external_mini_cluster.cc:1428] Setting key deb65219ff8619ef52baff1f2736b9a1
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:06:14.837704 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:14.824426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:36219 (local address 127.27.110.254:36341)
0812 07:06:14.824567 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:14.824572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:14.825702 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 07:06:14.827070 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:14.827108 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:14.827169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:14.827260 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:14.828970 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.829534 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.830847 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.831157 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.831272 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:14.833860 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:14.833863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:14.833866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:14.833891 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:14.835487 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:14.836035 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:14.836039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:14.836041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:14.836136 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:14.836502 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:14.836506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:14.836507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:14.836703 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:14.836810 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:14.837398 (+ 588us) server_negotiation.cc:299] Negotiation successful
0812 07:06:14.837532 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 07:06:14.838608 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:14.825003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36341 (local address 127.27.110.194:36219)
0812 07:06:14.825547 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:14.825594 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:14.826830 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:14.827504 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:14.827516 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:14.828037 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:14.828757 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:14.828775 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.829739 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.829743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:14.830700 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:14.830710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:14.831317 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:14.831321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:14.831505 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:14.832145 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:14.832163 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:14.833677 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:14.835617 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:14.835631 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:14.835647 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:14.835920 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:14.836240 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:14.836243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:14.836245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:14.836399 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:14.836827 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:14.836835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:14.837147 (+ 312us) client_negotiation.cc:769] Sending connection context
0812 07:06:14.837353 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 07:06:14.837730 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":149,"threads_started":1}
I20260812 07:06:14.839965 26416 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36341
I20260812 07:06:14.840269 26416 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:14.840797 26416 heartbeater.cc:511] Master 127.27.110.254:36341 requested a full tablet report, sending...
I20260812 07:06:14.841885 25834 ts_manager.cc:194] Registered new tserver with Master: ee4d4324fc0b43b997c31c7ac64941f1 (127.27.110.194:44021)
I20260812 07:06:14.842502 25834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:36219
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:06:14.854019 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36341
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:42125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:14.969910 26427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:14.970201 26427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:14.970325 26427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:14.974707 26427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:14.974828 26427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:14.974927 26427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:06:14.980430 26427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:42125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:36341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:14.981523 26427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:14.982589 26427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:14.992704 26432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.992683 26440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:14.992688 26435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:14.992911 26427 server_base.cc:1034] running on GCE node
I20260812 07:06:14.993294 26427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:14.993971 26427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:14.995170 26427 hybrid_clock.cc:648] HybridClock initialized: now 1786518374995141 us; error 29 us; skew 500 ppm
Aug 12 07:06:14 dist-test-slave-w4v5 krb5kdc[25752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:14.998119 26427 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:06:14.998921 26427 webserver.cc:460] Webserver started at http://127.27.110.195:39135/ using document root <none> and password file <none>
I20260812 07:06:14.999501 26427 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:14.999571 26427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:14.999737 26427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:15.001937 26427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "8b69641d449f48f1a7796d55edd0c77d"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "8704dccd01d23a2208b4d2007cb408c7"
server_key_iv: "5279729f18012581745eef8e9500e858"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.002453 26427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "8b69641d449f48f1a7796d55edd0c77d"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "8704dccd01d23a2208b4d2007cb408c7"
server_key_iv: "5279729f18012581745eef8e9500e858"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.006222 26427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:06:15.009048 26453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:15.010421 26427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:06:15.010540 26427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "8b69641d449f48f1a7796d55edd0c77d"
format_stamp: "Formatted at 2026-08-12 07:06:14 on dist-test-slave-w4v5"
server_key: "8704dccd01d23a2208b4d2007cb408c7"
server_key_iv: "5279729f18012581745eef8e9500e858"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.010658 26427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:15.061008 26427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:15.064684 26427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:15.064853 26427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:15.065459 26427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:15.066296 26427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:15.066346 26427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:15.066388 26427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:15.066403 26427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:15.076115 26427 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:33409
I20260812 07:06:15.076143 26663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:33409 every 8 connection(s)
I20260812 07:06:15.077103 26427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:15.082345 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 26427
I20260812 07:06:15.082508 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:15.082806 28091 external_mini_cluster.cc:1428] Setting key ad2ef6e72bf81008229ef82a569e22ed
Aug 12 07:06:15 dist-test-slave-w4v5 krb5kdc[25752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518374, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:15.092620 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.079056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:41589 (local address 127.27.110.254:36341)
0812 07:06:15.079260 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.079264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.080079 (+ 815us) server_negotiation.cc:407] Connection header received
0812 07:06:15.081510 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.081514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.081572 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.081687 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:15.083055 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.083717 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.085168 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.085398 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.085512 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.088597 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:15.088600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:15.088602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:15.088629 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:15.090298 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:15.090905 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:15.090909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:15.090911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:15.091012 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:15.091420 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:15.091424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:15.091425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:15.091640 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:15.091759 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.092309 (+ 550us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.092447 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 07:06:15.093493 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.079432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36341 (local address 127.27.110.195:41589)
0812 07:06:15.079916 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.079961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:15.081285 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:15.081873 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:15.081880 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:15.082280 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:15.082868 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.082906 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.083845 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.083848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.085045 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.085055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.085563 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.085567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.085744 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.086433 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:15.086452 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:15.088367 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:15.090462 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:15.090469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:15.090486 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:15.090787 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:15.091150 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:15.091153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:15.091155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:15.091314 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:15.091759 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:15.091767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:15.092043 (+ 276us) client_negotiation.cc:769] Sending connection context
0812 07:06:15.092247 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 07:06:15.092610 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20260812 07:06:15.094755 26664 heartbeater.cc:347] Connected to a master server at 127.27.110.254:36341
I20260812 07:06:15.095041 26664 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:15.095762 26664 heartbeater.cc:511] Master 127.27.110.254:36341 requested a full tablet report, sending...
I20260812 07:06:15.097018 25831 ts_manager.cc:194] Registered new tserver with Master: 8b69641d449f48f1a7796d55edd0c77d (127.27.110.195:33409)
I20260812 07:06:15.097674 25831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:41589
I20260812 07:06:15.109225 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:15.119772 25902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.111538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.27.110.254:36341)
0812 07:06:15.111935 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.111940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.112067 (+ 127us) server_negotiation.cc:407] Connection header received
0812 07:06:15.112233 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.112236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.112289 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.112378 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:15.113150 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.113705 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.114742 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.114982 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.115158 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.116593 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:15.116597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:15.116600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:15.116628 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:15.117889 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:15.118361 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:15.118365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:15.118367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:15.118456 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:15.118767 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:15.118770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:15.118772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:15.119013 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:15.119243 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.119472 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.119592 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260812 07:06:15.122948 25831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:15.125988 25831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:15.141042 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.135184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.27.110.194:44021)
0812 07:06:15.135656 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.135669 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.136082 (+ 413us) server_negotiation.cc:407] Connection header received
0812 07:06:15.136223 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.136228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.136407 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.136559 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:15.136990 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.138131 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.139507 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.140338 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.140473 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.140611 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.140787 (+ 176us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.140892 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":154,"threads_started":1}
I20260812 07:06:15.141029 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.134876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:44021 (local address 127.0.0.1:41414)
0812 07:06:15.135981 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.135992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:15.136097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:15.136727 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:15.136730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:15.136750 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:15.136862 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.136868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.138243 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.138246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.139384 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.139394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.140470 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.140473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.140570 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.140605 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:06:15.140738 (+ 133us) client_negotiation.cc:240] Negotiation successful
0812 07:06:15.140861 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1044,"mutex_wait_us":8,"thread_start_us":84,"threads_started":1}
I20260812 07:06:15.141103 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.134594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:33409 (local address 127.0.0.1:55668)
0812 07:06:15.135799 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.135812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:15.135932 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:15.136450 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:15.136459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:15.136488 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:15.136646 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.136654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.138160 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.138167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.139672 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.139684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.140746 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.140750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.140846 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.140866 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 07:06:15.140926 (+ 60us) client_negotiation.cc:240] Negotiation successful
0812 07:06:15.140982 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":148,"thread_start_us":122,"threads_started":1}
I20260812 07:06:15.141196 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.135389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.27.110.195:33409)
0812 07:06:15.135799 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.135818 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.135942 (+ 124us) server_negotiation.cc:407] Connection header received
0812 07:06:15.136059 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.136065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.136281 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.136443 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:15.136784 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.138035 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.139806 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.140618 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.140768 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.140891 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.140945 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.141021 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":113,"threads_started":1}
I20260812 07:06:15.143592 26686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.134535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39183 (local address 127.0.0.1:35182)
0812 07:06:15.135453 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.135512 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:15.135646 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:15.136644 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:15.136648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:15.136667 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:15.136798 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.136812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.137793 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.137800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.139672 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.139684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.143149 (+ 3465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.143152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.143252 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.143276 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 07:06:15.143371 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 07:06:15.143442 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":105,"thread_start_us":97,"threads_started":1}
I20260812 07:06:15.143702 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.134680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35182 (local address 127.27.110.193:39183)
0812 07:06:15.136189 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.136196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.136216 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:06:15.136286 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.136292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.136455 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.136664 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:15.136931 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.137676 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.141384 (+ 3708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.142259 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.142371 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.142456 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.143422 (+ 966us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.143522 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1408,"thread_start_us":84,"threads_started":1}
I20260812 07:06:15.144824 26298 tablet_service.cc:1467] Processing CreateTablet for tablet 4979b704ef0c40bbbeee10bd5ffb6080 (DEFAULT_TABLE table=test-table [id=1c6c3ccc4a2b43eba1a3d6408ee36933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:15.145009 26544 tablet_service.cc:1467] Processing CreateTablet for tablet 4979b704ef0c40bbbeee10bd5ffb6080 (DEFAULT_TABLE table=test-table [id=1c6c3ccc4a2b43eba1a3d6408ee36933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:15.145936 26298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979b704ef0c40bbbeee10bd5ffb6080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:15.146051 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979b704ef0c40bbbeee10bd5ffb6080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:15.146639 26058 tablet_service.cc:1467] Processing CreateTablet for tablet 4979b704ef0c40bbbeee10bd5ffb6080 (DEFAULT_TABLE table=test-table [id=1c6c3ccc4a2b43eba1a3d6408ee36933]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:15.147630 26058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4979b704ef0c40bbbeee10bd5ffb6080. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:15.152645 26694 tablet_bootstrap.cc:492] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Bootstrap starting.
I20260812 07:06:15.153270 26695 tablet_bootstrap.cc:492] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Bootstrap starting.
I20260812 07:06:15.154235 26697 tablet_bootstrap.cc:492] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: Bootstrap starting.
I20260812 07:06:15.155095 26694 tablet_bootstrap.cc:654] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:15.155421 26695 tablet_bootstrap.cc:654] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:15.156291 26697 tablet_bootstrap.cc:654] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:15.156798 26695 log.cc:824] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:15.156798 26694 log.cc:824] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:15.157145 26697 log.cc:824] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:15.158743 26695 tablet_bootstrap.cc:492] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: No bootstrap required, opened a new log
I20260812 07:06:15.158891 26694 tablet_bootstrap.cc:492] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: No bootstrap required, opened a new log
I20260812 07:06:15.158969 26695 ts_tablet_manager.cc:1397] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:06:15.159099 26697 tablet_bootstrap.cc:492] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: No bootstrap required, opened a new log
I20260812 07:06:15.159099 26694 ts_tablet_manager.cc:1397] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:06:15.159276 26697 ts_tablet_manager.cc:1397] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260812 07:06:15.162194 26695 raft_consensus.cc:348] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.162563 26695 raft_consensus.cc:374] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:15.162647 26695 raft_consensus.cc:729] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b69641d449f48f1a7796d55edd0c77d, State: Initialized, Role: FOLLOWER
I20260812 07:06:15.163142 26695 consensus_queue.cc:260] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.163133 26694 raft_consensus.cc:348] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.163403 26694 raft_consensus.cc:374] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:15.163496 26694 raft_consensus.cc:729] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee4d4324fc0b43b997c31c7ac64941f1, State: Initialized, Role: FOLLOWER
I20260812 07:06:15.163393 26697 raft_consensus.cc:348] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.163698 26697 raft_consensus.cc:374] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:15.163785 26664 heartbeater.cc:503] Master 127.27.110.254:36341 was elected leader, sending a full tablet report...
I20260812 07:06:15.163798 26697 raft_consensus.cc:729] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270d8fbc57964fc1ae345f94c13d758a, State: Initialized, Role: FOLLOWER
I20260812 07:06:15.163965 26694 consensus_queue.cc:260] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.164124 26695 ts_tablet_manager.cc:1428] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260812 07:06:15.164201 26697 consensus_queue.cc:260] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.164700 26416 heartbeater.cc:503] Master 127.27.110.254:36341 was elected leader, sending a full tablet report...
I20260812 07:06:15.164880 26694 ts_tablet_manager.cc:1428] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:06:15.164989 26697 ts_tablet_manager.cc:1428] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:06:15.165277 26177 heartbeater.cc:503] Master 127.27.110.254:36341 was elected leader, sending a full tablet report...
W20260812 07:06:15.323954 26419 tablet.cc:2369] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:06:15.328218 26666 tablet.cc:2369] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 07:06:15.337564 26179 tablet.cc:2369] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:15.359990 26706 raft_consensus.cc:482] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:15.360266 26706 raft_consensus.cc:504] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.361626 26706 leader_election.cc:288] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 270d8fbc57964fc1ae345f94c13d758a (127.27.110.193:39183), ee4d4324fc0b43b997c31c7ac64941f1 (127.27.110.194:44021)
I20260812 07:06:15.366063 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.362025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:55063 (local address 127.27.110.193:39183)
0812 07:06:15.362163 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.362168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.362187 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:06:15.362364 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.362368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.362433 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.362515 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:15.362897 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.363539 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.364848 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.365328 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.365424 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.365467 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.365777 (+ 310us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.365886 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 07:06:15.366094 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.362134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:43317 (local address 127.27.110.194:44021)
0812 07:06:15.362279 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.362285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.362451 (+ 166us) server_negotiation.cc:407] Connection header received
0812 07:06:15.362644 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.362648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.362712 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.362811 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:15.363179 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.363816 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.364991 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.365462 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.365606 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.365667 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.365849 (+ 182us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.365937 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:06:15.366135 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.361905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39183 (local address 127.27.110.195:55063)
0812 07:06:15.362098 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.362116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:15.362226 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:15.362548 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:15.362552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:15.362591 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:15.362738 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.362749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.363666 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.363670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.364690 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.364700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.365456 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.365460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.365591 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.365652 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 07:06:15.365766 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:06:15.365875 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":26}
I20260812 07:06:15.366163 26714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.361934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:44021 (local address 127.27.110.195:43317)
0812 07:06:15.362353 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.362366 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:15.362469 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:15.362884 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:15.362887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:15.362908 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:15.363022 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.363029 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.363969 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.363972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.364852 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:15.364863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.365588 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.365591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:15.365685 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.365724 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 07:06:15.365819 (+ 95us) client_negotiation.cc:240] Negotiation successful
0812 07:06:15.365886 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":2,"thread_start_us":137,"threads_started":1}
I20260812 07:06:15.366796 26333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979b704ef0c40bbbeee10bd5ffb6080" candidate_uuid: "8b69641d449f48f1a7796d55edd0c77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" is_pre_election: true
I20260812 07:06:15.366814 26094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979b704ef0c40bbbeee10bd5ffb6080" candidate_uuid: "8b69641d449f48f1a7796d55edd0c77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270d8fbc57964fc1ae345f94c13d758a" is_pre_election: true
I20260812 07:06:15.367156 26333 raft_consensus.cc:2454] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b69641d449f48f1a7796d55edd0c77d in term 0.
I20260812 07:06:15.367187 26094 raft_consensus.cc:2454] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b69641d449f48f1a7796d55edd0c77d in term 0.
I20260812 07:06:15.367719 26455 leader_election.cc:302] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 270d8fbc57964fc1ae345f94c13d758a, 8b69641d449f48f1a7796d55edd0c77d, ee4d4324fc0b43b997c31c7ac64941f1; no voters:
I20260812 07:06:15.368016 26706 raft_consensus.cc:2781] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:15.368105 26706 raft_consensus.cc:482] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:15.368150 26706 raft_consensus.cc:3037] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:15.369186 26706 raft_consensus.cc:504] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.369518 26706 leader_election.cc:288] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [CANDIDATE]: Term 1 election: Requested vote from peers 270d8fbc57964fc1ae345f94c13d758a (127.27.110.193:39183), ee4d4324fc0b43b997c31c7ac64941f1 (127.27.110.194:44021)
I20260812 07:06:15.369887 26333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979b704ef0c40bbbeee10bd5ffb6080" candidate_uuid: "8b69641d449f48f1a7796d55edd0c77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee4d4324fc0b43b997c31c7ac64941f1"
I20260812 07:06:15.369916 26094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4979b704ef0c40bbbeee10bd5ffb6080" candidate_uuid: "8b69641d449f48f1a7796d55edd0c77d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270d8fbc57964fc1ae345f94c13d758a"
I20260812 07:06:15.369997 26333 raft_consensus.cc:3037] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:15.370064 26094 raft_consensus.cc:3037] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:15.371148 26333 raft_consensus.cc:2454] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b69641d449f48f1a7796d55edd0c77d in term 1.
I20260812 07:06:15.371466 26094 raft_consensus.cc:2454] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b69641d449f48f1a7796d55edd0c77d in term 1.
I20260812 07:06:15.371467 26455 leader_election.cc:302] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b69641d449f48f1a7796d55edd0c77d, ee4d4324fc0b43b997c31c7ac64941f1; no voters:
I20260812 07:06:15.371706 26706 raft_consensus.cc:2781] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:15.371984 26706 raft_consensus.cc:686] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 1 LEADER]: Becoming Leader. State: Replica: 8b69641d449f48f1a7796d55edd0c77d, State: Running, Role: LEADER
I20260812 07:06:15.372375 26706 consensus_queue.cc:237] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } }
I20260812 07:06:15.375958 25831 catalog_manager.cc:5495] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b69641d449f48f1a7796d55edd0c77d (127.27.110.195). New cstate: current_term: 1 leader_uuid: "8b69641d449f48f1a7796d55edd0c77d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b69641d449f48f1a7796d55edd0c77d" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 33409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:15.438274 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.434905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43622 (local address 127.27.110.195:33409)
0812 07:06:15.435104 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.435108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.435121 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:06:15.435201 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.435203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.435262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.435352 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:15.435593 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.436089 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.436972 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.437187 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.437279 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.437538 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:15.437946 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:15.438047 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.438091 (+ 44us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.438142 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 07:06:15.444257 26333 raft_consensus.cc:1261] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 1 FOLLOWER]: Refusing update from remote peer 8b69641d449f48f1a7796d55edd0c77d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:15.444257 26094 raft_consensus.cc:1261] T 4979b704ef0c40bbbeee10bd5ffb6080 P 270d8fbc57964fc1ae345f94c13d758a [term 1 FOLLOWER]: Refusing update from remote peer 8b69641d449f48f1a7796d55edd0c77d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:15.445017 26716 consensus_queue.cc:1035] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee4d4324fc0b43b997c31c7ac64941f1" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 44021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:15.445206 26706 consensus_queue.cc:1035] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [LEADER]: Connected to new peer: Peer: permanent_uuid: "270d8fbc57964fc1ae345f94c13d758a" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:15.452714 26721 mvcc.cc:205] Tried to move back new op lower bound from 7317579265812054016 to 7317579265528586240. Current Snapshot: MvccSnapshot[applied={T|T < 7317579265812054016}]
I20260812 07:06:15.455446 26724 mvcc.cc:205] Tried to move back new op lower bound from 7317579265812054016 to 7317579265528586240. Current Snapshot: MvccSnapshot[applied={T|T < 7317579265812054016}]
I20260812 07:06:15.462146 25831 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:15.462301 25831 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:15.464721 25831 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce4366a01c5f4fe4a0d5c1f818342bdc: Sending DeleteTablet for 3 replicas of tablet 4979b704ef0c40bbbeee10bd5ffb6080
I20260812 07:06:15.465487 26544 tablet_service.cc:1514] Processing DeleteTablet for tablet 4979b704ef0c40bbbeee10bd5ffb6080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:55668
I20260812 07:06:15.465550 26058 tablet_service.cc:1514] Processing DeleteTablet for tablet 4979b704ef0c40bbbeee10bd5ffb6080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:35182
I20260812 07:06:15.466089 26298 tablet_service.cc:1514] Processing DeleteTablet for tablet 4979b704ef0c40bbbeee10bd5ffb6080 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:15 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:41414
I20260812 07:06:15.466167 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 25947
I20260812 07:06:15.469410 26740 tablet_replica.cc:330] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: stopping tablet replica
I20260812 07:06:15.469775 26740 raft_consensus.cc:2229] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:15.470219 26740 raft_consensus.cc:2258] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:15.471711 26743 tablet_replica.cc:330] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: stopping tablet replica
I20260812 07:06:15.472041 26743 raft_consensus.cc:2229] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:15.472388 26743 raft_consensus.cc:2258] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:15.474788 26740 ts_tablet_manager.cc:1905] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:15.478106 26740 ts_tablet_manager.cc:1918] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:15.478288 26740 log.cc:1196] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/4979b704ef0c40bbbeee10bd5ffb6080
I20260812 07:06:15.478847 26740 ts_tablet_manager.cc:1939] T 4979b704ef0c40bbbeee10bd5ffb6080 P 8b69641d449f48f1a7796d55edd0c77d: Deleting consensus metadata
I20260812 07:06:15.479547 26743 ts_tablet_manager.cc:1905] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:15.480391 25810 catalog_manager.cc:4841] TS 8b69641d449f48f1a7796d55edd0c77d (127.27.110.195:33409): tablet 4979b704ef0c40bbbeee10bd5ffb6080 (table test-table [id=1c6c3ccc4a2b43eba1a3d6408ee36933]) successfully deleted
I20260812 07:06:15.483994 26743 ts_tablet_manager.cc:1918] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:15.484215 26743 log.cc:1196] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/4979b704ef0c40bbbeee10bd5ffb6080
I20260812 07:06:15.484655 26743 ts_tablet_manager.cc:1939] T 4979b704ef0c40bbbeee10bd5ffb6080 P ee4d4324fc0b43b997c31c7ac64941f1: Deleting consensus metadata
I20260812 07:06:15.485963 25808 catalog_manager.cc:4841] TS ee4d4324fc0b43b997c31c7ac64941f1 (127.27.110.194:44021): tablet 4979b704ef0c40bbbeee10bd5ffb6080 (table test-table [id=1c6c3ccc4a2b43eba1a3d6408ee36933]) successfully deleted
W20260812 07:06:15.496966 25809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:39183 (error 108)
I20260812 07:06:15.497071 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 26187
I20260812 07:06:15.501354 26686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.501010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:39183 (local address 127.0.0.1:59014)
0812 07:06:15.501143 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:15.501233 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:39183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 07:06:15.501540 25809 catalog_manager.cc:4584] TS 270d8fbc57964fc1ae345f94c13d758a (127.27.110.193:39183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4979b704ef0c40bbbeee10bd5ffb6080: Network error: Client connection negotiation failed: client connection to 127.27.110.193:39183: connect: Connection refused (error 111)
I20260812 07:06:15.510422 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 26427
I20260812 07:06:15.527729 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 25779
2026-08-12T07:06:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2941 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T07:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:15Z Disabled control of system clock
I20260812 07:06:15.565219 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44793
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:06:15.678165 26750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:15.678416 26750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:15.678449 26750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:15.682559 26750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:15.682616 26750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:15.682632 26750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:15.682648 26750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:15.682663 26750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:15.687325 26750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:44793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:15.688374 26750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:15.689754 26750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:15.698103 26756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:15.698120 26760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:15.698096 26757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:15.698202 26750 server_base.cc:1034] running on GCE node
I20260812 07:06:15.698892 26750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:15.700026 26750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:15.701196 26750 hybrid_clock.cc:648] HybridClock initialized: now 1786518375701167 us; error 41 us; skew 500 ppm
I20260812 07:06:15.707836 26750 webserver.cc:460] Webserver started at http://127.27.110.254:33097/ using document root <none> and password file <none>
I20260812 07:06:15.708463 26750 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:15.708531 26750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:15.708783 26750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:15.710947 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "92a85ed6d9834ce3ac19c1e42d964fce"
format_stamp: "Formatted at 2026-08-12 07:06:15 on dist-test-slave-w4v5"
server_key: "a298d1fcc303b033d7e62da77ea5fd9c"
server_key_iv: "ec5da8e1ce796237699add70b27c9f7c"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.711592 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "92a85ed6d9834ce3ac19c1e42d964fce"
format_stamp: "Formatted at 2026-08-12 07:06:15 on dist-test-slave-w4v5"
server_key: "a298d1fcc303b033d7e62da77ea5fd9c"
server_key_iv: "ec5da8e1ce796237699add70b27c9f7c"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.715384 26750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:15.717733 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:15.719017 26750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:15.719208 26750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "92a85ed6d9834ce3ac19c1e42d964fce"
format_stamp: "Formatted at 2026-08-12 07:06:15 on dist-test-slave-w4v5"
server_key: "a298d1fcc303b033d7e62da77ea5fd9c"
server_key_iv: "ec5da8e1ce796237699add70b27c9f7c"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.719328 26750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:15.731113 26750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:15.731805 26750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:15.732010 26750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:15.739441 26750 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:44793
I20260812 07:06:15.739465 26872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:44793 every 8 connection(s)
I20260812 07:06:15.740399 26750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:15.741264 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 26750
I20260812 07:06:15.741523 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:15.741813 28091 external_mini_cluster.cc:1428] Setting key 88b2fbd6e9299a19fdcc078d548fd7b6
I20260812 07:06:15.744093 26873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:15.749544 26873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce: Bootstrap starting.
I20260812 07:06:15.751312 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:15.743297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.27.110.254:44793)
0812 07:06:15.743957 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:15.743976 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:15.744013 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:06:15.744741 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:15.744760 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:15.745164 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:15.745507 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:15.745941 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.746911 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.747902 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:15.748372 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:15.748551 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:15.749464 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:15.749469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:15.749482 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:15.749925 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:15.749981 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:15.750002 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:15.750110 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:15.750271 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 07:06:15.750535 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":151,"threads_started":1}
I20260812 07:06:15.751992 26873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:15.752774 26873 log.cc:824] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:15.754706 26873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce: No bootstrap required, opened a new log
I20260812 07:06:15.758553 26873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44793 } }
I20260812 07:06:15.758780 26873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:15.758859 26873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92a85ed6d9834ce3ac19c1e42d964fce, State: Initialized, Role: FOLLOWER
I20260812 07:06:15.759341 26873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44793 } }
I20260812 07:06:15.759465 26873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:15.759552 26873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:15.759652 26873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:15.760676 26873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44793 } }
I20260812 07:06:15.761034 26873 leader_election.cc:302] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92a85ed6d9834ce3ac19c1e42d964fce; no voters:
I20260812 07:06:15.761358 26873 leader_election.cc:288] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:15.761571 26878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:15.761829 26878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [term 1 LEADER]: Becoming Leader. State: Replica: 92a85ed6d9834ce3ac19c1e42d964fce, State: Running, Role: LEADER
I20260812 07:06:15.762144 26878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44793 } }
I20260812 07:06:15.762569 26873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:15.763813 26879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44793 } } }
I20260812 07:06:15.763945 26879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:15.764238 26880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92a85ed6d9834ce3ac19c1e42d964fce. Latest consensus state: current_term: 1 leader_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92a85ed6d9834ce3ac19c1e42d964fce" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44793 } } }
I20260812 07:06:15.764331 26898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:15.764353 26880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:15.767236 26898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:15.772739 26898 catalog_manager.cc:1347] Generated new cluster ID: 23c0f57822f446d8acdb1a97062ddaba
I20260812 07:06:15.772801 26898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:15.783555 26898 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:15.784578 26898 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:15.789954 26898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce: Generated new TSK 0
I20260812 07:06:15.790807 26898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:15.804260 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44793
--builtin_ntp_servers=127.27.110.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:06:15.926539 26916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:15.926818 26916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:15.926857 26916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:15.930927 26916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:15.930981 26916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:15.931037 26916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:06:15.935784 26916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:15.936818 26916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:15.938201 26916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:06:15.947957 26916 server_base.cc:1034] running on GCE node
W20260812 07:06:15.947911 26927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:15.947925 26929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:15.947911 26935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:15.948607 26916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:15.949304 26916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:15.950479 26916 hybrid_clock.cc:648] HybridClock initialized: now 1786518375950464 us; error 28 us; skew 500 ppm
I20260812 07:06:15.952427 26916 webserver.cc:460] Webserver started at http://127.27.110.193:37471/ using document root <none> and password file <none>
I20260812 07:06:15.953037 26916 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:15.953109 26916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:15.953341 26916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:15.955458 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "c724d8e6ddf041018bc2bc7c1cfe5574"
format_stamp: "Formatted at 2026-08-12 07:06:15 on dist-test-slave-w4v5"
server_key: "f858d2137847e8f022253b93d707d276"
server_key_iv: "c6b0171dbcbe8106e18b5a775cc77791"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.956030 26916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "c724d8e6ddf041018bc2bc7c1cfe5574"
format_stamp: "Formatted at 2026-08-12 07:06:15 on dist-test-slave-w4v5"
server_key: "f858d2137847e8f022253b93d707d276"
server_key_iv: "c6b0171dbcbe8106e18b5a775cc77791"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.959755 26916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:06:15.962158 26944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:15.963574 26916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260812 07:06:15.963701 26916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "c724d8e6ddf041018bc2bc7c1cfe5574"
format_stamp: "Formatted at 2026-08-12 07:06:15 on dist-test-slave-w4v5"
server_key: "f858d2137847e8f022253b93d707d276"
server_key_iv: "c6b0171dbcbe8106e18b5a775cc77791"
server_key_version: "encryptionkey@0"
I20260812 07:06:15.963807 26916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:15.997174 26916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:15.997931 26916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:15.998149 26916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:15.998777 26916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:15.999758 26916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:15.999832 26916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:15.999904 26916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:15.999956 26916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.015499 26916 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:40605
I20260812 07:06:16.015535 27143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:40605 every 8 connection(s)
I20260812 07:06:16.017536 26916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:16.024111 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 26916
I20260812 07:06:16.024328 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:16.024849 28091 external_mini_cluster.cc:1428] Setting key d272f839526dc2da080f11b9fd2df85c
I20260812 07:06:16.028823 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44793
--builtin_ntp_servers=127.27.110.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:06:16.037269 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.020903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:53647 (local address 127.27.110.254:44793)
0812 07:06:16.021211 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.021215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.023456 (+ 2241us) server_negotiation.cc:407] Connection header received
0812 07:06:16.026857 (+ 3401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.026861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.026944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.027153 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:16.029682 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.030688 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.033368 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.033762 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.033979 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.036169 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:16.036185 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:16.036188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:16.036213 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:16.036242 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:16.036263 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:16.036414 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.036718 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.036907 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 07:06:16.039289 27150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.021783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:44793 (local address 127.27.110.193:53647)
0812 07:06:16.023164 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.023268 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.026506 (+ 3238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.027587 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.027609 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.027846 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:16.029350 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.029404 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.030991 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.030999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.033140 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.033159 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.034070 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.034077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.034393 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.035639 (+ 1246us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:16.035641 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:16.035729 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:16.035854 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:16.036415 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:16.036440 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:16.036451 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.036689 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.037369 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":242,"threads_started":1}
I20260812 07:06:16.041755 27146 heartbeater.cc:347] Connected to a master server at 127.27.110.254:44793
I20260812 07:06:16.042142 27146 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:16.043596 27146 heartbeater.cc:511] Master 127.27.110.254:44793 requested a full tablet report, sending...
I20260812 07:06:16.047680 26803 ts_manager.cc:194] Registered new tserver with Master: c724d8e6ddf041018bc2bc7c1cfe5574 (127.27.110.193:40605)
I20260812 07:06:16.051201 26803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.193:53647
W20260812 07:06:16.260922 27152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:16.261391 27152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:16.261512 27152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:16.268652 27152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:16.268872 27152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:16.269018 27152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:06:16.277329 27152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:16.279263 27152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:16.281679 27152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:16.298807 27163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:16.298844 27162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:16.299026 27168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:16.299118 27152 server_base.cc:1034] running on GCE node
I20260812 07:06:16.300213 27152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:16.301298 27152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:16.304741 27152 hybrid_clock.cc:648] HybridClock initialized: now 1786518376304668 us; error 94 us; skew 500 ppm
I20260812 07:06:16.308409 27152 webserver.cc:460] Webserver started at http://127.27.110.194:33917/ using document root <none> and password file <none>
I20260812 07:06:16.309595 27152 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:16.309801 27152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:16.310215 27152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:16.314111 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "99f6d7edc62b41bdb630aee9d32e5468"
format_stamp: "Formatted at 2026-08-12 07:06:16 on dist-test-slave-w4v5"
server_key: "8ebc96460b8b09a6fac3c8ee9ed07471"
server_key_iv: "5690711ac4c6d6f41146092c694a356b"
server_key_version: "encryptionkey@0"
I20260812 07:06:16.315416 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "99f6d7edc62b41bdb630aee9d32e5468"
format_stamp: "Formatted at 2026-08-12 07:06:16 on dist-test-slave-w4v5"
server_key: "8ebc96460b8b09a6fac3c8ee9ed07471"
server_key_iv: "5690711ac4c6d6f41146092c694a356b"
server_key_version: "encryptionkey@0"
I20260812 07:06:16.322791 27152 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 07:06:16.327755 27178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.330111 27152 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 07:06:16.330361 27152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "99f6d7edc62b41bdb630aee9d32e5468"
format_stamp: "Formatted at 2026-08-12 07:06:16 on dist-test-slave-w4v5"
server_key: "8ebc96460b8b09a6fac3c8ee9ed07471"
server_key_iv: "5690711ac4c6d6f41146092c694a356b"
server_key_version: "encryptionkey@0"
I20260812 07:06:16.330606 27152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:16.363514 27152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:16.364882 27152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:16.365250 27152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:16.366451 27152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:16.368508 27152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:16.368635 27152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.368806 27152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:16.368875 27152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.386036 27152 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:36467
I20260812 07:06:16.386073 27403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:36467 every 8 connection(s)
I20260812 07:06:16.387825 27152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:16.394093 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 27152
I20260812 07:06:16.394227 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:16.394526 28091 external_mini_cluster.cc:1428] Setting key a496bc6c21a1238cd0e9e2c4b4fa5e5b
I20260812 07:06:16.396752 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44793
--builtin_ntp_servers=127.27.110.212:45127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:06:16.400358 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.391119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:53565 (local address 127.27.110.254:44793)
0812 07:06:16.391361 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.391365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.392647 (+ 1282us) server_negotiation.cc:407] Connection header received
0812 07:06:16.394293 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.394298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.394366 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.394446 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:16.395633 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.396320 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.398125 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.398344 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.398428 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.399761 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:16.399767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:16.399771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:16.399788 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:16.399813 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:16.399826 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:16.399918 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.400064 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.400142 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 07:06:16.401114 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.391760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:44793 (local address 127.27.110.194:53565)
0812 07:06:16.392475 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.392518 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.394003 (+ 1485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.394680 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.394689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.394799 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:16.395481 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.395497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.396461 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.396468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.397997 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.398012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.398506 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.398513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.398769 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.399474 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:16.399476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:16.399516 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:16.399601 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:16.399913 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:16.399920 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:16.399924 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.400015 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.400234 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":128,"threads_started":1}
I20260812 07:06:16.402357 27405 heartbeater.cc:347] Connected to a master server at 127.27.110.254:44793
I20260812 07:06:16.402590 27405 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:16.403173 27405 heartbeater.cc:511] Master 127.27.110.254:44793 requested a full tablet report, sending...
I20260812 07:06:16.404280 26803 ts_manager.cc:194] Registered new tserver with Master: 99f6d7edc62b41bdb630aee9d32e5468 (127.27.110.194:36467)
I20260812 07:06:16.404857 26803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.194:53565
W20260812 07:06:16.514518 27409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:16.514780 27409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:16.514819 27409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:16.518877 27409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:16.518939 27409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:16.519029 27409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:06:16.524123 27409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:45127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:44793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:16.525267 27409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:16.526717 27409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:16.536819 27417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:16.536819 27418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:16.536839 27422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:16.537094 27409 server_base.cc:1034] running on GCE node
I20260812 07:06:16.537665 27409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:16.538368 27409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:16.539649 27409 hybrid_clock.cc:648] HybridClock initialized: now 1786518376539621 us; error 36 us; skew 500 ppm
I20260812 07:06:16.541658 27409 webserver.cc:460] Webserver started at http://127.27.110.195:41549/ using document root <none> and password file <none>
I20260812 07:06:16.542230 27409 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:16.542292 27409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:16.542464 27409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:16.544608 27409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "ff8787bb384b444c8ac57718fe42ef53"
format_stamp: "Formatted at 2026-08-12 07:06:16 on dist-test-slave-w4v5"
server_key: "29e2ea50bd3c663c35077c636ffa23e7"
server_key_iv: "65eaf8b3a2d6b580e15bf408a8236bc3"
server_key_version: "encryptionkey@0"
I20260812 07:06:16.545145 27409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "ff8787bb384b444c8ac57718fe42ef53"
format_stamp: "Formatted at 2026-08-12 07:06:16 on dist-test-slave-w4v5"
server_key: "29e2ea50bd3c663c35077c636ffa23e7"
server_key_iv: "65eaf8b3a2d6b580e15bf408a8236bc3"
server_key_version: "encryptionkey@0"
I20260812 07:06:16.548874 27409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:06:16.551343 27431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.552675 27409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:16.552806 27409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "ff8787bb384b444c8ac57718fe42ef53"
format_stamp: "Formatted at 2026-08-12 07:06:16 on dist-test-slave-w4v5"
server_key: "29e2ea50bd3c663c35077c636ffa23e7"
server_key_iv: "65eaf8b3a2d6b580e15bf408a8236bc3"
server_key_version: "encryptionkey@0"
I20260812 07:06:16.552919 27409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:16.573693 27409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:16.574802 27409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:16.575050 27409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:16.575829 27409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:16.576862 27409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:16.576911 27409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.576984 27409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:16.577018 27409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:16.586664 27409 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:39447
I20260812 07:06:16.586828 27641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:39447 every 8 connection(s)
I20260812 07:06:16.587735 27409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:16.595228 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 27409
I20260812 07:06:16.595367 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:16.595675 28091 external_mini_cluster.cc:1428] Setting key 03c8c07a97164c161f2d564945d009cd
I20260812 07:06:16.597704 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.589795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:41233 (local address 127.27.110.254:44793)
0812 07:06:16.589976 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.589996 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.591099 (+ 1103us) server_negotiation.cc:407] Connection header received
0812 07:06:16.592469 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.592473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.592523 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.592615 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:16.593516 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.594064 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.595507 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.595813 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.595900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.597208 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:16.597213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:16.597216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:16.597231 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:16.597255 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:16.597265 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:16.597340 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.597503 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.597563 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 07:06:16.598568 27644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.590194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:44793 (local address 127.27.110.195:41233)
0812 07:06:16.590924 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.590966 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.592248 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.592794 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.592801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.592884 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:16.593382 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.593393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.594177 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.594179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.595307 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.595319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.595975 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.595980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.596170 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.596877 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:16.596879 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:16.596935 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:16.597021 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:16.597357 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:16.597366 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:16.597371 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.597478 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.597739 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260812 07:06:16.599686 27642 heartbeater.cc:347] Connected to a master server at 127.27.110.254:44793
I20260812 07:06:16.599931 27642 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:16.600474 27642 heartbeater.cc:511] Master 127.27.110.254:44793 requested a full tablet report, sending...
I20260812 07:06:16.601466 26803 ts_manager.cc:194] Registered new tserver with Master: ff8787bb384b444c8ac57718fe42ef53 (127.27.110.195:39447)
I20260812 07:06:16.602046 26803 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.195:41233
I20260812 07:06:16.609778 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:16.616025 26876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.611887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55950 (local address 127.27.110.254:44793)
0812 07:06:16.612053 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.612058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.612244 (+ 186us) server_negotiation.cc:407] Connection header received
0812 07:06:16.612327 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.612329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.612368 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.612435 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:16.612884 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.613418 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.614334 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.614563 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.614647 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.615523 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:16.615526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:16.615528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:16.615540 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:16.615558 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:16.615566 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:16.615628 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.615788 (+ 160us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.615836 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:06:16.619328 26803 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:16.622414 26803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:16.636839 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.631165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.27.110.194:36467)
0812 07:06:16.632186 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.632199 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.632226 (+ 27us) server_negotiation.cc:407] Connection header received
0812 07:06:16.632316 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.632322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.632550 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.633030 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:16.633185 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.634076 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.635581 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.636137 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.636284 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.636478 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.636565 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.636676 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":904,"thread_start_us":165,"threads_started":1}
I20260812 07:06:16.636842 27662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.630950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36467 (local address 127.0.0.1:38158)
0812 07:06:16.631352 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.631408 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.631543 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.632767 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.632772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.632791 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:16.632905 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.632909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.634196 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.634202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.635443 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.635453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.636267 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.636271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.636381 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.636419 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.636536 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.636661 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":82,"threads_started":1}
I20260812 07:06:16.638279 27669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.632155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40605 (local address 127.0.0.1:54024)
0812 07:06:16.632511 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.632550 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.632641 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.633166 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.633170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.633187 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:16.633366 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.633373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.634836 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.634842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.636257 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.636268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.637695 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.637700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.637806 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.637842 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.638032 (+ 190us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.638120 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":132,"spinlock_wait_cycles":12160,"thread_start_us":94,"threads_started":1}
I20260812 07:06:16.638736 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.632325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.27.110.193:40605)
0812 07:06:16.632716 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.632723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.632748 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:06:16.632829 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.632834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.633017 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.633177 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:16.633500 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.634691 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.636406 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.637450 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.637596 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.637762 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.638468 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.638589 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":103,"threads_started":1}
I20260812 07:06:16.638594 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.631334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:39447 (local address 127.0.0.1:32782)
0812 07:06:16.631816 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.631830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.631913 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.633181 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.633184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.633205 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:16.633330 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.633337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.634856 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.634858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.636261 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.636268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.638219 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.638222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.638317 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.638348 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.638410 (+ 62us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.638477 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":128,"thread_start_us":127,"threads_started":1}
I20260812 07:06:16.639307 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.631602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.27.110.195:39447)
0812 07:06:16.632719 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.632740 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.632759 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:06:16.632829 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.632834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.633017 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.633500 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:16.633647 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.634734 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.636406 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.638004 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.638725 (+ 721us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.638919 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.638977 (+ 58us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.639125 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":192,"threads_started":1}
I20260812 07:06:16.640473 27279 tablet_service.cc:1467] Processing CreateTablet for tablet daaddf12d22d4ed980ca7756287984d5 (DEFAULT_TABLE table=test-table [id=daf73e17dc874f9d8d3ed09e733a7494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:16.641486 27279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaddf12d22d4ed980ca7756287984d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:16.641892 27039 tablet_service.cc:1467] Processing CreateTablet for tablet daaddf12d22d4ed980ca7756287984d5 (DEFAULT_TABLE table=test-table [id=daf73e17dc874f9d8d3ed09e733a7494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:16.642977 27039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaddf12d22d4ed980ca7756287984d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:16.642829 27524 tablet_service.cc:1467] Processing CreateTablet for tablet daaddf12d22d4ed980ca7756287984d5 (DEFAULT_TABLE table=test-table [id=daf73e17dc874f9d8d3ed09e733a7494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:16.643848 27524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daaddf12d22d4ed980ca7756287984d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:16.648072 27671 tablet_bootstrap.cc:492] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Bootstrap starting.
I20260812 07:06:16.650224 27672 tablet_bootstrap.cc:492] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: Bootstrap starting.
I20260812 07:06:16.650394 27671 tablet_bootstrap.cc:654] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:16.651341 27673 tablet_bootstrap.cc:492] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Bootstrap starting.
I20260812 07:06:16.651688 27671 log.cc:824] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:16.652547 27672 tablet_bootstrap.cc:654] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:16.653374 27672 log.cc:824] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:16.653384 27673 tablet_bootstrap.cc:654] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:16.654193 27673 log.cc:824] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:16.654573 27671 tablet_bootstrap.cc:492] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: No bootstrap required, opened a new log
I20260812 07:06:16.654807 27671 ts_tablet_manager.cc:1397] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260812 07:06:16.655341 27672 tablet_bootstrap.cc:492] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: No bootstrap required, opened a new log
I20260812 07:06:16.655540 27672 ts_tablet_manager.cc:1397] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:16.656353 27673 tablet_bootstrap.cc:492] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: No bootstrap required, opened a new log
I20260812 07:06:16.656571 27673 ts_tablet_manager.cc:1397] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:16.659173 27672 raft_consensus.cc:348] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.659448 27672 raft_consensus.cc:374] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:16.659512 27672 raft_consensus.cc:729] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c724d8e6ddf041018bc2bc7c1cfe5574, State: Initialized, Role: FOLLOWER
I20260812 07:06:16.659624 27671 raft_consensus.cc:348] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.659878 27671 raft_consensus.cc:374] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:16.659940 27671 raft_consensus.cc:729] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6d7edc62b41bdb630aee9d32e5468, State: Initialized, Role: FOLLOWER
I20260812 07:06:16.660004 27672 consensus_queue.cc:260] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.660357 27671 consensus_queue.cc:260] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.660787 27673 raft_consensus.cc:348] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.661020 27673 raft_consensus.cc:374] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:16.661075 27673 raft_consensus.cc:729] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff8787bb384b444c8ac57718fe42ef53, State: Initialized, Role: FOLLOWER
I20260812 07:06:16.661091 27405 heartbeater.cc:503] Master 127.27.110.254:44793 was elected leader, sending a full tablet report...
I20260812 07:06:16.661335 27671 ts_tablet_manager.cc:1428] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 07:06:16.661463 27673 consensus_queue.cc:260] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.662153 27642 heartbeater.cc:503] Master 127.27.110.254:44793 was elected leader, sending a full tablet report...
I20260812 07:06:16.662317 27672 ts_tablet_manager.cc:1428] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:06:16.662349 27673 ts_tablet_manager.cc:1428] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:06:16.662616 27146 heartbeater.cc:503] Master 127.27.110.254:44793 was elected leader, sending a full tablet report...
I20260812 07:06:16.719058 27685 raft_consensus.cc:482] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:16.719430 27685 raft_consensus.cc:504] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.720660 27685 leader_election.cc:288] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff8787bb384b444c8ac57718fe42ef53 (127.27.110.195:39447), c724d8e6ddf041018bc2bc7c1cfe5574 (127.27.110.193:40605)
I20260812 07:06:16.725742 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.721094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:38991 (local address 127.27.110.193:40605)
0812 07:06:16.721230 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.721235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.721645 (+ 410us) server_negotiation.cc:407] Connection header received
0812 07:06:16.721762 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.721765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.721838 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.721920 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:16.722165 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.722917 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.724332 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.724942 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.725083 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.725134 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.725528 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.725606 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:06:16.725742 27668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.721112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:49525 (local address 127.27.110.195:39447)
0812 07:06:16.721291 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.721297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.721317 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:06:16.721390 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.721397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.721456 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.721527 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:16.721867 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.722669 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.724332 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.724942 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.725081 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.725134 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.725528 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.725626 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 07:06:16.725770 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.720969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:39447 (local address 127.27.110.194:49525)
0812 07:06:16.721137 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.721157 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.721270 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.721573 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.721577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.721597 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:16.721734 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.721744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.722788 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.722791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.724162 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.724174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.725102 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.725109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.725372 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.725407 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.725491 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.725645 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":105}
I20260812 07:06:16.725780 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.720969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40605 (local address 127.27.110.194:38991)
0812 07:06:16.721494 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.721528 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:16.721664 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:16.721935 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:16.721938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:16.721953 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:16.722048 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.722052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.723008 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.723011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.724190 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:16.724197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.725122 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.725125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:16.725343 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.725405 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 07:06:16.725491 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 07:06:16.725624 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":266,"thread_start_us":142,"threads_started":1}
I20260812 07:06:16.726653 27558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaddf12d22d4ed980ca7756287984d5" candidate_uuid: "99f6d7edc62b41bdb630aee9d32e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8787bb384b444c8ac57718fe42ef53" is_pre_election: true
I20260812 07:06:16.726650 27069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaddf12d22d4ed980ca7756287984d5" candidate_uuid: "99f6d7edc62b41bdb630aee9d32e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" is_pre_election: true
I20260812 07:06:16.726997 27558 raft_consensus.cc:2454] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6d7edc62b41bdb630aee9d32e5468 in term 0.
I20260812 07:06:16.726997 27069 raft_consensus.cc:2454] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6d7edc62b41bdb630aee9d32e5468 in term 0.
I20260812 07:06:16.727551 27190 leader_election.cc:302] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6d7edc62b41bdb630aee9d32e5468, c724d8e6ddf041018bc2bc7c1cfe5574; no voters:
I20260812 07:06:16.727902 27685 raft_consensus.cc:2781] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:16.727991 27685 raft_consensus.cc:482] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:16.728041 27685 raft_consensus.cc:3037] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:16.729161 27685 raft_consensus.cc:504] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.729573 27685 leader_election.cc:288] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [CANDIDATE]: Term 1 election: Requested vote from peers ff8787bb384b444c8ac57718fe42ef53 (127.27.110.195:39447), c724d8e6ddf041018bc2bc7c1cfe5574 (127.27.110.193:40605)
I20260812 07:06:16.730000 27558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaddf12d22d4ed980ca7756287984d5" candidate_uuid: "99f6d7edc62b41bdb630aee9d32e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff8787bb384b444c8ac57718fe42ef53"
I20260812 07:06:16.730126 27069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daaddf12d22d4ed980ca7756287984d5" candidate_uuid: "99f6d7edc62b41bdb630aee9d32e5468" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574"
I20260812 07:06:16.730129 27558 raft_consensus.cc:3037] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:16.730296 27069 raft_consensus.cc:3037] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:16.731472 27558 raft_consensus.cc:2454] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6d7edc62b41bdb630aee9d32e5468 in term 1.
I20260812 07:06:16.731691 27069 raft_consensus.cc:2454] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6d7edc62b41bdb630aee9d32e5468 in term 1.
I20260812 07:06:16.731839 27179 leader_election.cc:302] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6d7edc62b41bdb630aee9d32e5468, ff8787bb384b444c8ac57718fe42ef53; no voters:
I20260812 07:06:16.732023 27685 raft_consensus.cc:2781] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:16.732283 27685 raft_consensus.cc:686] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 1 LEADER]: Becoming Leader. State: Replica: 99f6d7edc62b41bdb630aee9d32e5468, State: Running, Role: LEADER
I20260812 07:06:16.732627 27685 consensus_queue.cc:237] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } }
I20260812 07:06:16.735874 26803 catalog_manager.cc:5495] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6d7edc62b41bdb630aee9d32e5468 (127.27.110.194). New cstate: current_term: 1 leader_uuid: "99f6d7edc62b41bdb630aee9d32e5468" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6d7edc62b41bdb630aee9d32e5468" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:16.762203 27663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.756635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.27.110.194:36467)
0812 07:06:16.756996 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:16.757001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:16.757027 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:06:16.757171 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:16.757174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:16.757268 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:16.757397 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:16.757801 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.758316 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.759899 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:16.760348 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:16.760486 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:16.760907 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:16.761570 (+ 663us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:16.761696 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:16.761810 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 07:06:16.761880 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 07:06:16.769516 27147 tablet.cc:2369] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:16.769723 27069 raft_consensus.cc:1261] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 1 FOLLOWER]: Refusing update from remote peer 99f6d7edc62b41bdb630aee9d32e5468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:16.769744 27558 raft_consensus.cc:1261] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 1 FOLLOWER]: Refusing update from remote peer 99f6d7edc62b41bdb630aee9d32e5468: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:16.770527 27685 consensus_queue.cc:1035] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff8787bb384b444c8ac57718fe42ef53" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 39447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:16.770741 27692 consensus_queue.cc:1035] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c724d8e6ddf041018bc2bc7c1cfe5574" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:16.786154 26800 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:16.786347 26800 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:16.789099 26800 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92a85ed6d9834ce3ac19c1e42d964fce: Sending DeleteTablet for 3 replicas of tablet daaddf12d22d4ed980ca7756287984d5
I20260812 07:06:16.790042 27039 tablet_service.cc:1514] Processing DeleteTablet for tablet daaddf12d22d4ed980ca7756287984d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:16 UTC) from {username='slave'} at 127.0.0.1:54024
I20260812 07:06:16.790030 27524 tablet_service.cc:1514] Processing DeleteTablet for tablet daaddf12d22d4ed980ca7756287984d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:16 UTC) from {username='slave'} at 127.0.0.1:32782
I20260812 07:06:16.790190 27279 tablet_service.cc:1514] Processing DeleteTablet for tablet daaddf12d22d4ed980ca7756287984d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:16 UTC) from {username='slave'} at 127.0.0.1:38158
I20260812 07:06:16.790611 27722 tablet_replica.cc:330] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574: stopping tablet replica
I20260812 07:06:16.790834 27724 tablet_replica.cc:330] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: stopping tablet replica
I20260812 07:06:16.790977 27722 raft_consensus.cc:2229] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:16.791208 27724 raft_consensus.cc:2229] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:16.791350 27722 raft_consensus.cc:2258] T daaddf12d22d4ed980ca7756287984d5 P c724d8e6ddf041018bc2bc7c1cfe5574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:16.791479 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 26916
I20260812 07:06:16.794950 27724 raft_consensus.cc:2258] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:16.795502 27719 tablet_replica.cc:330] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: stopping tablet replica
I20260812 07:06:16.795989 27719 raft_consensus.cc:2229] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:16.796438 27719 raft_consensus.cc:2258] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:16.802556 27724 ts_tablet_manager.cc:1905] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:16.805929 27724 ts_tablet_manager.cc:1918] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:16.806030 27724 log.cc:1196] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/daaddf12d22d4ed980ca7756287984d5
I20260812 07:06:16.806327 27724 ts_tablet_manager.cc:1939] T daaddf12d22d4ed980ca7756287984d5 P 99f6d7edc62b41bdb630aee9d32e5468: Deleting consensus metadata
I20260812 07:06:16.807880 26775 catalog_manager.cc:4841] TS 99f6d7edc62b41bdb630aee9d32e5468 (127.27.110.194:36467): tablet daaddf12d22d4ed980ca7756287984d5 (table test-table [id=daf73e17dc874f9d8d3ed09e733a7494]) successfully deleted
I20260812 07:06:16.825276 27719 ts_tablet_manager.cc:1905] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:16.828364 27719 ts_tablet_manager.cc:1918] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:16.828517 27719 log.cc:1196] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/daaddf12d22d4ed980ca7756287984d5
W20260812 07:06:16.828737 26779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:40605 (error 108)
I20260812 07:06:16.828835 27719 ts_tablet_manager.cc:1939] T daaddf12d22d4ed980ca7756287984d5 P ff8787bb384b444c8ac57718fe42ef53: Deleting consensus metadata
I20260812 07:06:16.829964 27669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:16.829610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40605 (local address 127.0.0.1:54026)
0812 07:06:16.829753 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:16.829842 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 07:06:16.830107 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 27152
W20260812 07:06:16.830333 26779 catalog_manager.cc:4584] TS c724d8e6ddf041018bc2bc7c1cfe5574 (127.27.110.193:40605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet daaddf12d22d4ed980ca7756287984d5: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40605: connect: Connection refused (error 111)
I20260812 07:06:16.833076 26772 catalog_manager.cc:4841] TS ff8787bb384b444c8ac57718fe42ef53 (127.27.110.195:39447): tablet daaddf12d22d4ed980ca7756287984d5 (table test-table [id=daf73e17dc874f9d8d3ed09e733a7494]) successfully deleted
I20260812 07:06:16.848876 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 27409
I20260812 07:06:16.865046 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 26750
2026-08-12T07:06:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1337 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:16 dist-test-slave-w4v5 krb5kdc[27729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:16 dist-test-slave-w4v5 krb5kdc[27729](info): set up 2 sockets
Aug 12 07:06:16 dist-test-slave-w4v5 krb5kdc[27729](info): commencing operation
krb5kdc: starting...
W20260812 07:06:18.167532 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.274s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:18 dist-test-slave-w4v5 krb5kdc[27729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:18.301645 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:34635
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:34635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:18.414906 27747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:18.415264 27747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:18.415354 27747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:18.419389 27747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:18.419476 27747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:18.419513 27747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:18.419549 27747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:18.419626 27747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:18.424924 27747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:34635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:34635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:18.426064 27747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:18.427129 27747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:18.435665 27757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:18.435613 27759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:18.435882 27755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:18.435767 27747 server_base.cc:1034] running on GCE node
I20260812 07:06:18.436668 27747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:18.438035 27747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:18.439224 27747 hybrid_clock.cc:648] HybridClock initialized: now 1786518378439201 us; error 32 us; skew 500 ppm
Aug 12 07:06:18 dist-test-slave-w4v5 krb5kdc[27729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:18.442133 27747 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:18.443034 27747 webserver.cc:460] Webserver started at http://127.27.110.254:45141/ using document root <none> and password file <none>
I20260812 07:06:18.443645 27747 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:18.443742 27747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:18.443982 27747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:18.446033 27747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "ab639bed8b6045528aa16519ccbf6678"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "d5608e63f82538bfb5af07e390b7deb7"
server_key_iv: "bdffeec5ed47c379566e973f2634a170"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.446606 27747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "ab639bed8b6045528aa16519ccbf6678"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "d5608e63f82538bfb5af07e390b7deb7"
server_key_iv: "bdffeec5ed47c379566e973f2634a170"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.450613 27747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 07:06:18.453096 27774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:18.454290 27747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:18.454434 27747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "ab639bed8b6045528aa16519ccbf6678"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "d5608e63f82538bfb5af07e390b7deb7"
server_key_iv: "bdffeec5ed47c379566e973f2634a170"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.454550 27747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:18.479985 27747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:18.483393 27747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:18.483621 27747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:18.491854 27747 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:34635
I20260812 07:06:18.491880 27884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:34635 every 8 connection(s)
I20260812 07:06:18.492909 27747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:18.496068 27888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:18.498885 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 27747
I20260812 07:06:18.499055 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:18.499372 28091 external_mini_cluster.cc:1428] Setting key ff4aa449d20f12959f852dc9ba9df49d
I20260812 07:06:18.503046 27888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678: Bootstrap starting.
I20260812 07:06:18.505936 27888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:18.506860 27888 log.cc:824] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:18.508803 27888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678: No bootstrap required, opened a new log
Aug 12 07:06:18 dist-test-slave-w4v5 krb5kdc[27729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:18.513139 27888 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab639bed8b6045528aa16519ccbf6678" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34635 } }
I20260812 07:06:18.513417 27888 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:18.513469 27888 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab639bed8b6045528aa16519ccbf6678, State: Initialized, Role: FOLLOWER
I20260812 07:06:18.513996 27888 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab639bed8b6045528aa16519ccbf6678" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34635 } }
I20260812 07:06:18.514194 27888 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:18.514269 27888 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:18.514381 27888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:18.515589 27888 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab639bed8b6045528aa16519ccbf6678" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34635 } }
I20260812 07:06:18.515935 27888 leader_election.cc:302] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab639bed8b6045528aa16519ccbf6678; no voters:
I20260812 07:06:18.516279 27888 leader_election.cc:288] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:18.516431 27895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:18.516726 27895 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [term 1 LEADER]: Becoming Leader. State: Replica: ab639bed8b6045528aa16519ccbf6678, State: Running, Role: LEADER
I20260812 07:06:18.516944 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:18.500731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.27.110.254:34635)
0812 07:06:18.501169 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:18.501191 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:18.501236 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:06:18.502162 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:18.502188 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:18.502626 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:18.503032 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:18.503875 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:18.505529 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:18.506496 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:18.507312 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:18.507496 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:18.510554 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:18.510561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:18.510585 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:18.510616 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:18.513436 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:18.513978 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:18.513983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:18.513989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:18.514162 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:18.514526 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:18.514529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:18.514531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:18.515030 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:18.515273 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:18.515571 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 07:06:18.515885 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":121,"threads_started":1}
I20260812 07:06:18.517086 27895 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab639bed8b6045528aa16519ccbf6678" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34635 } }
I20260812 07:06:18.517537 27888 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:18.518862 27896 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab639bed8b6045528aa16519ccbf6678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab639bed8b6045528aa16519ccbf6678" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34635 } } }
I20260812 07:06:18.518992 27896 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:18.519342 27898 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab639bed8b6045528aa16519ccbf6678. Latest consensus state: current_term: 1 leader_uuid: "ab639bed8b6045528aa16519ccbf6678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab639bed8b6045528aa16519ccbf6678" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 34635 } } }
I20260812 07:06:18.519423 27910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:18.519459 27898 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:18.522861 27910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:18.528396 27910 catalog_manager.cc:1347] Generated new cluster ID: 922f6fdf7d474c25a5f6f29f0a38a35a
I20260812 07:06:18.528478 27910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:18.548777 27910 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:18.549906 27910 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:18.555714 27910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678: Generated new TSK 0
I20260812 07:06:18.556460 27910 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:06:18.598533 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34635
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:18.723733 27931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:18.724062 27931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:18.724121 27931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:18.729269 27931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:18.729363 27931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:18.729432 27931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:06:18.735909 27931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:18.737123 27931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:18.738366 27931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:18.749214 27946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:18.749197 27938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:18.749197 27940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:18.749568 27931 server_base.cc:1034] running on GCE node
I20260812 07:06:18.750036 27931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:18.750752 27931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:18.751941 27931 hybrid_clock.cc:648] HybridClock initialized: now 1786518378751890 us; error 62 us; skew 500 ppm
Aug 12 07:06:18 dist-test-slave-w4v5 krb5kdc[27729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:18.755054 27931 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:06:18.756115 27931 webserver.cc:460] Webserver started at http://127.27.110.193:38703/ using document root <none> and password file <none>
I20260812 07:06:18.756768 27931 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:18.756875 27931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:18.757156 27931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:18.759997 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "6bc3220cfb03948921d0e0fd79a900e1"
server_key_iv: "76876a1f1430fc9af6d0970b60034732"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.760612 27931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "6bc3220cfb03948921d0e0fd79a900e1"
server_key_iv: "76876a1f1430fc9af6d0970b60034732"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.766398 27931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 07:06:18.769438 27964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:18.770850 27931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 07:06:18.770958 27931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "6bc3220cfb03948921d0e0fd79a900e1"
server_key_iv: "76876a1f1430fc9af6d0970b60034732"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.771049 27931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:18.782886 27931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:18.786247 27931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:18.786448 27931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:18.787029 27931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:18.787909 27931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:18.787956 27931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:18.787997 27931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:18.788012 27931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:18.798029 27931 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:46777
I20260812 07:06:18.798074 28192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:46777 every 8 connection(s)
I20260812 07:06:18.798949 27931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:18.807039 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 27931
I20260812 07:06:18.807240 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:18.807588 28091 external_mini_cluster.cc:1428] Setting key 41e90826d129bea30bfacad753832acb
Aug 12 07:06:18 dist-test-slave-w4v5 krb5kdc[27729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:18.814769 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:18.800845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:35147 (local address 127.27.110.254:34635)
0812 07:06:18.801006 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:18.801011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:18.801968 (+ 957us) server_negotiation.cc:407] Connection header received
0812 07:06:18.803306 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:18.803309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:18.803356 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:18.803433 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:18.804635 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:18.805203 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:18.806409 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:18.806686 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:18.806771 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:18.810179 (+ 3408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:18.810182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:18.810184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:18.810213 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:18.812155 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:18.812841 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:18.812845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:18.812847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:18.812944 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:18.813354 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:18.813362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:18.813364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:18.813593 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:18.813712 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:18.814467 (+ 755us) server_negotiation.cc:299] Negotiation successful
0812 07:06:18.814622 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:06:18.816164 28210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:18.801292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:34635 (local address 127.27.110.193:35147)
0812 07:06:18.801807 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:18.801852 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:18.803093 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:18.803594 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:18.803627 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:18.803970 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:18.804446 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:18.804460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:18.805338 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:18.805341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:18.806278 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:18.806289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:18.806856 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:18.806862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:18.807101 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:18.807915 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:18.807937 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:18.809978 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:18.812320 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:18.812328 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:18.812347 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:18.812697 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:18.813062 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:18.813065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:18.813068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:18.813246 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:18.813711 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:18.813720 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:18.814092 (+ 372us) client_negotiation.cc:769] Sending connection context
0812 07:06:18.814347 (+ 255us) client_negotiation.cc:240] Negotiation successful
0812 07:06:18.814798 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":123,"threads_started":1}
I20260812 07:06:18.817803 28194 heartbeater.cc:347] Connected to a master server at 127.27.110.254:34635
I20260812 07:06:18.818126 28194 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:18.818724 28194 heartbeater.cc:511] Master 127.27.110.254:34635 requested a full tablet report, sending...
I20260812 07:06:18.820541 27817 ts_manager.cc:194] Registered new tserver with Master: f2b6b88d5f1d41c3902e9444bf4f48c9 (127.27.110.193:46777)
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:06:18.821789 27817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:35147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:06:18.843042 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34635
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:18.961580 28214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:18.961863 28214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:18.961959 28214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:18.965981 28214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:18.966063 28214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:18.966156 28214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:06:18.971601 28214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:18.973133 28214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:18.974222 28214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:06:18.984464 28214 server_base.cc:1034] running on GCE node
W20260812 07:06:18.984424 28221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:18.984416 28222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:18.984412 28228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:18.985153 28214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:18.985788 28214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:18.987020 28214 hybrid_clock.cc:648] HybridClock initialized: now 1786518378986989 us; error 46 us; skew 500 ppm
Aug 12 07:06:18 dist-test-slave-w4v5 krb5kdc[27729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:18.989940 28214 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:06:18.991333 28214 webserver.cc:460] Webserver started at http://127.27.110.194:43971/ using document root <none> and password file <none>
I20260812 07:06:18.991899 28214 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:18.991961 28214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:18.992136 28214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:18.994207 28214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "34c9f0a8747a481fb3d34ec974ae4ad2"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "84e6c9042b52912c0c9003ad3bd96732"
server_key_iv: "d2fbd48f931c201acc84547d9269e90c"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.994757 28214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "34c9f0a8747a481fb3d34ec974ae4ad2"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "84e6c9042b52912c0c9003ad3bd96732"
server_key_iv: "d2fbd48f931c201acc84547d9269e90c"
server_key_version: "encryptionkey@0"
I20260812 07:06:18.998639 28214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260812 07:06:19.001217 28239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.002437 28214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:06:19.002533 28214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "34c9f0a8747a481fb3d34ec974ae4ad2"
format_stamp: "Formatted at 2026-08-12 07:06:18 on dist-test-slave-w4v5"
server_key: "84e6c9042b52912c0c9003ad3bd96732"
server_key_iv: "d2fbd48f931c201acc84547d9269e90c"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.002645 28214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:19.018697 28214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:19.022535 28214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:19.022706 28214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:19.023361 28214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:19.024202 28214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:19.024255 28214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.024299 28214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:19.024317 28214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.035223 28214 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:36289
I20260812 07:06:19.035249 28463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:36289 every 8 connection(s)
I20260812 07:06:19.036214 28214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:19.041567 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28214
I20260812 07:06:19.041729 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:19.042049 28091 external_mini_cluster.cc:1428] Setting key aecce32e0178bb0626ba298711f34d18
Aug 12 07:06:19 dist-test-slave-w4v5 krb5kdc[27729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518378, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:19.054026 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.038301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:38883 (local address 127.27.110.254:34635)
0812 07:06:19.038452 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.038455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.039442 (+ 987us) server_negotiation.cc:407] Connection header received
0812 07:06:19.040938 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.040941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.041005 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.041116 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:19.042804 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.043433 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.045259 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.045539 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.045635 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.049021 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:19.049029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:19.049035 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:19.049075 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:19.051258 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:19.051953 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:19.051957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:19.051958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:19.052125 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:19.052588 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:19.052593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:19.052595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:19.052899 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:19.053040 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.053602 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.053761 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:06:19.054852 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.038693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:34635 (local address 127.27.110.194:38883)
0812 07:06:19.039279 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.039327 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.040706 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.041385 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.041398 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.041955 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:19.042640 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.042658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.043563 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.043567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.045065 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.045080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.045712 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.045719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.045968 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.046722 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:19.046738 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:19.048812 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:19.051517 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:19.051523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:19.051535 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:19.051809 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:19.052264 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:19.052267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:19.052268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:19.052425 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:19.053054 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:19.053060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:19.053309 (+ 249us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.053493 (+ 184us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.053776 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260812 07:06:19.056226 28464 heartbeater.cc:347] Connected to a master server at 127.27.110.254:34635
I20260812 07:06:19.056459 28464 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:19.057010 28464 heartbeater.cc:511] Master 127.27.110.254:34635 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:06:19.058331 27817 ts_manager.cc:194] Registered new tserver with Master: 34c9f0a8747a481fb3d34ec974ae4ad2 (127.27.110.194:36289)
I20260812 07:06:19.058938 27817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:38883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:06:19.074432 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34635
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:19.196085 28471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:19.196377 28471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:19.196463 28471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:19.200549 28471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:19.200630 28471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:19.200737 28471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:06:19.206147 28471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:34635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:19.207293 28471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:19.208321 28471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:19.218940 28483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:19.218899 28482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:19.219151 28471 server_base.cc:1034] running on GCE node
W20260812 07:06:19.218909 28488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:19.219645 28471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:19.220291 28471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:19.221465 28471 hybrid_clock.cc:648] HybridClock initialized: now 1786518379221448 us; error 27 us; skew 500 ppm
Aug 12 07:06:19 dist-test-slave-w4v5 krb5kdc[27729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:19.224434 28471 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:06:19.225369 28471 webserver.cc:460] Webserver started at http://127.27.110.195:37883/ using document root <none> and password file <none>
I20260812 07:06:19.225889 28471 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:19.225948 28471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:19.226114 28471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:19.228181 28471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "5687d30f8c674415a2e8d8ae3ee105fb"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "abd089ffb3712248b489f622f0f0f511"
server_key_iv: "baa90a17f325625bb3a80939651edb55"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.228761 28471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "5687d30f8c674415a2e8d8ae3ee105fb"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "abd089ffb3712248b489f622f0f0f511"
server_key_iv: "baa90a17f325625bb3a80939651edb55"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.232587 28471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:19.235361 28498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.236755 28471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:19.236910 28471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "5687d30f8c674415a2e8d8ae3ee105fb"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "abd089ffb3712248b489f622f0f0f511"
server_key_iv: "baa90a17f325625bb3a80939651edb55"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.237037 28471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:19.246907 28471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:19.250298 28471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:19.250526 28471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:19.251207 28471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:19.252116 28471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:19.252183 28471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.252255 28471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:19.252306 28471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.262207 28471 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:40781
I20260812 07:06:19.262238 28712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:40781 every 8 connection(s)
I20260812 07:06:19.263214 28471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:19.272454 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28471
I20260812 07:06:19.272632 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:19.272936 28091 external_mini_cluster.cc:1428] Setting key 81faa3d5995b08629ea3dc08dadadf3b
Aug 12 07:06:19 dist-test-slave-w4v5 krb5kdc[27729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518379, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:19.278584 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.265278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:39443 (local address 127.27.110.254:34635)
0812 07:06:19.265506 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.265509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.266371 (+ 862us) server_negotiation.cc:407] Connection header received
0812 07:06:19.267852 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.267855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.267904 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.267985 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:19.269280 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.269829 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.271153 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.271389 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.271481 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.274657 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:19.274661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:19.274664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:19.274701 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:19.276401 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:19.276950 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:19.276953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:19.276954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:19.277061 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:19.277450 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:19.277453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:19.277454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:19.277688 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:19.277781 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.278309 (+ 528us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.278420 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260812 07:06:19.279481 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.265686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:34635 (local address 127.27.110.195:39443)
0812 07:06:19.266206 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.266253 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.267629 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.268209 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.268217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.268594 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:19.269116 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.269131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.269972 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.269975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.270934 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.270945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.271584 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.271588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.271751 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.272444 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:19.272462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:19.274480 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:19.276561 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:19.276567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:19.276580 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:19.276854 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:19.277173 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:19.277176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:19.277177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:19.277330 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:19.277803 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:19.277809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:19.278057 (+ 248us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.278228 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.278500 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":138,"threads_started":1}
I20260812 07:06:19.280900 28713 heartbeater.cc:347] Connected to a master server at 127.27.110.254:34635
I20260812 07:06:19.281201 28713 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:19.281780 28713 heartbeater.cc:511] Master 127.27.110.254:34635 requested a full tablet report, sending...
I20260812 07:06:19.282931 27817 ts_manager.cc:194] Registered new tserver with Master: 5687d30f8c674415a2e8d8ae3ee105fb (127.27.110.195:40781)
I20260812 07:06:19.283592 27817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:39443
I20260812 07:06:19.287745 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:19.298640 27891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.290025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.27.110.254:34635)
0812 07:06:19.290211 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.290218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.290307 (+ 89us) server_negotiation.cc:407] Connection header received
0812 07:06:19.290483 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.290486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.290534 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.290632 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:19.291454 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.291989 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.292897 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.293138 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.293259 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.295058 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:19.295062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:19.295065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:19.295124 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:19.296613 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:19.297192 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:19.297196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:19.297198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:19.297288 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:19.297648 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:19.297652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:19.297654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:19.297896 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:19.298065 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.298355 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.298459 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 07:06:19.302121 27817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:19.305419 27817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:19.321106 28741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.314409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36289 (local address 127.0.0.1:57020)
0812 07:06:19.315522 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.315535 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.315641 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.316200 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.316207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.316230 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.316383 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.316390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.317980 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.317983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.319687 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.319698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.320530 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.320533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.320637 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.320673 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.320783 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.320905 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":348,"thread_start_us":96,"threads_started":1}
I20260812 07:06:19.321249 28739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.314118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:46777 (local address 127.0.0.1:55732)
0812 07:06:19.315030 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.315114 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.315251 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.316383 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.316386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.316402 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.316509 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.316516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.317862 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.317868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.319697 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.319708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.320807 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.320811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.320901 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.320921 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.321006 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.321068 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":116,"spinlock_wait_cycles":13440,"thread_start_us":114,"threads_started":1}
I20260812 07:06:19.321419 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.314144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:40781 (local address 127.0.0.1:50948)
0812 07:06:19.315030 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.315114 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.315251 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.316603 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.316606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.316625 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.316729 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.316736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.318304 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.318307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.319697 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.319708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.321007 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.321010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.321108 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.321131 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.321206 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.321291 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":71,"spinlock_wait_cycles":16896,"thread_start_us":99,"threads_started":1}
I20260812 07:06:19.321776 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.314859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.27.110.194:36289)
0812 07:06:19.315536 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.315546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.315605 (+ 59us) server_negotiation.cc:407] Connection header received
0812 07:06:19.315766 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.315771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.316022 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.316212 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.316530 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.317766 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.319809 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.320380 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.320539 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.320679 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.321519 (+ 840us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.321633 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":150,"threads_started":1}
I20260812 07:06:19.321939 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.315038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.27.110.195:40781)
0812 07:06:19.316073 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.316096 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.316116 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:06:19.316182 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.316188 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.316355 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.316479 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.316868 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.318002 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.319846 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.320693 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.320814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.320936 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.321687 (+ 751us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.321786 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":142,"threads_started":1}
I20260812 07:06:19.322204 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.314283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.27.110.193:46777)
0812 07:06:19.315896 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.315915 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.315941 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:06:19.316019 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.316024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.316213 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.316414 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.316652 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.317748 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.319846 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.320693 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.320815 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.320936 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.321037 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.321133 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1500,"thread_start_us":165,"threads_started":1}
I20260812 07:06:19.324755 28058 tablet_service.cc:1467] Processing CreateTablet for tablet ad06f80fc9d6429caa6e8361ffebde1c (DEFAULT_TABLE table=test-table [id=1ac4c1de6cd546c6b980a71a42a8b7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:19.324858 28331 tablet_service.cc:1467] Processing CreateTablet for tablet ad06f80fc9d6429caa6e8361ffebde1c (DEFAULT_TABLE table=test-table [id=1ac4c1de6cd546c6b980a71a42a8b7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:19.324880 28591 tablet_service.cc:1467] Processing CreateTablet for tablet ad06f80fc9d6429caa6e8361ffebde1c (DEFAULT_TABLE table=test-table [id=1ac4c1de6cd546c6b980a71a42a8b7e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:19.325809 28058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad06f80fc9d6429caa6e8361ffebde1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:19.325917 28591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad06f80fc9d6429caa6e8361ffebde1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:19.325917 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad06f80fc9d6429caa6e8361ffebde1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:19.333026 28745 tablet_bootstrap.cc:492] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9: Bootstrap starting.
I20260812 07:06:19.333629 28748 tablet_bootstrap.cc:492] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Bootstrap starting.
I20260812 07:06:19.334208 28747 tablet_bootstrap.cc:492] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Bootstrap starting.
I20260812 07:06:19.335479 28745 tablet_bootstrap.cc:654] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:19.335826 28748 tablet_bootstrap.cc:654] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:19.336004 28747 tablet_bootstrap.cc:654] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:19.336478 28745 log.cc:824] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:19.337103 28748 log.cc:824] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:19.337196 28747 log.cc:824] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:19.338721 28745 tablet_bootstrap.cc:492] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9: No bootstrap required, opened a new log
I20260812 07:06:19.338932 28745 ts_tablet_manager.cc:1397] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:06:19.339150 28747 tablet_bootstrap.cc:492] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: No bootstrap required, opened a new log
I20260812 07:06:19.339346 28747 ts_tablet_manager.cc:1397] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:19.339390 28748 tablet_bootstrap.cc:492] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: No bootstrap required, opened a new log
I20260812 07:06:19.339558 28748 ts_tablet_manager.cc:1397] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:06:19.342553 28747 raft_consensus.cc:348] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.342801 28747 raft_consensus.cc:374] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:19.342855 28747 raft_consensus.cc:729] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5687d30f8c674415a2e8d8ae3ee105fb, State: Initialized, Role: FOLLOWER
I20260812 07:06:19.343312 28747 consensus_queue.cc:260] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.343552 28745 raft_consensus.cc:348] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.343552 28748 raft_consensus.cc:348] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.343809 28748 raft_consensus.cc:374] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:19.343809 28745 raft_consensus.cc:374] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:19.343873 28748 raft_consensus.cc:729] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34c9f0a8747a481fb3d34ec974ae4ad2, State: Initialized, Role: FOLLOWER
I20260812 07:06:19.343873 28745 raft_consensus.cc:729] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b6b88d5f1d41c3902e9444bf4f48c9, State: Initialized, Role: FOLLOWER
I20260812 07:06:19.344147 28713 heartbeater.cc:503] Master 127.27.110.254:34635 was elected leader, sending a full tablet report...
I20260812 07:06:19.344286 28745 consensus_queue.cc:260] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.344286 28748 consensus_queue.cc:260] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.345093 28194 heartbeater.cc:503] Master 127.27.110.254:34635 was elected leader, sending a full tablet report...
I20260812 07:06:19.345498 28747 ts_tablet_manager.cc:1428] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260812 07:06:19.346189 28745 ts_tablet_manager.cc:1428] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260812 07:06:19.346789 28748 ts_tablet_manager.cc:1428] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:06:19.347164 28464 heartbeater.cc:503] Master 127.27.110.254:34635 was elected leader, sending a full tablet report...
I20260812 07:06:19.433678 28754 raft_consensus.cc:482] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:19.433878 28754 raft_consensus.cc:504] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.435113 28754 leader_election.cc:288] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2b6b88d5f1d41c3902e9444bf4f48c9 (127.27.110.193:46777), 34c9f0a8747a481fb3d34ec974ae4ad2 (127.27.110.194:36289)
I20260812 07:06:19.439674 28716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.435312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:46777 (local address 127.27.110.195:55477)
0812 07:06:19.435459 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.435476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.435599 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.436040 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.436042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.436059 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.436230 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.436237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.437197 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.437200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.438185 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.438203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.439216 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.439220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.439322 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.439340 (+ 18us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.439404 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.439534 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":99}
I20260812 07:06:19.439747 28740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.435462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:55477 (local address 127.27.110.193:46777)
0812 07:06:19.435585 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.435590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.435608 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:06:19.435773 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.435776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.435840 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.435921 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.436342 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.437079 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.438354 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.438957 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.439100 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.439156 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.439478 (+ 322us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.439602 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260812 07:06:19.439775 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.435640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:46749 (local address 127.27.110.194:36289)
0812 07:06:19.435794 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.435798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.435864 (+ 66us) server_negotiation.cc:407] Connection header received
0812 07:06:19.436037 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.436043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.436103 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.436185 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.436486 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.437114 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.438301 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.438957 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.439102 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.439156 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.439582 (+ 426us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.439667 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 07:06:19.440018 28755 raft_consensus.cc:482] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:19.440143 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.435407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36289 (local address 127.27.110.195:46749)
0812 07:06:19.435780 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.435791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.435919 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.436214 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.436217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.436232 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.436371 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.436376 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.437229 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.437234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.438168 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.438183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.439217 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.439221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.439325 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.439346 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.439903 (+ 557us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.439983 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":5,"thread_start_us":153,"threads_started":1}
I20260812 07:06:19.440186 28755 raft_consensus.cc:504] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.440337 28104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad06f80fc9d6429caa6e8361ffebde1c" candidate_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" is_pre_election: true
I20260812 07:06:19.440693 28104 raft_consensus.cc:2454] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5687d30f8c674415a2e8d8ae3ee105fb in term 0.
I20260812 07:06:19.440882 28372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad06f80fc9d6429caa6e8361ffebde1c" candidate_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" is_pre_election: true
I20260812 07:06:19.441174 28372 raft_consensus.cc:2454] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5687d30f8c674415a2e8d8ae3ee105fb in term 0.
I20260812 07:06:19.441309 28509 leader_election.cc:302] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5687d30f8c674415a2e8d8ae3ee105fb, f2b6b88d5f1d41c3902e9444bf4f48c9; no voters:
I20260812 07:06:19.441603 28754 raft_consensus.cc:2781] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:19.441697 28754 raft_consensus.cc:482] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:19.441741 28754 raft_consensus.cc:3037] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:19.442185 28755 leader_election.cc:288] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5687d30f8c674415a2e8d8ae3ee105fb (127.27.110.195:40781), 34c9f0a8747a481fb3d34ec974ae4ad2 (127.27.110.194:36289)
I20260812 07:06:19.443322 28754 raft_consensus.cc:504] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.443673 28754 leader_election.cc:288] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [CANDIDATE]: Term 1 election: Requested vote from peers f2b6b88d5f1d41c3902e9444bf4f48c9 (127.27.110.193:46777), 34c9f0a8747a481fb3d34ec974ae4ad2 (127.27.110.194:36289)
I20260812 07:06:19.444084 28372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad06f80fc9d6429caa6e8361ffebde1c" candidate_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2"
I20260812 07:06:19.444084 28104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad06f80fc9d6429caa6e8361ffebde1c" candidate_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9"
I20260812 07:06:19.444218 28104 raft_consensus.cc:3037] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:19.444226 28372 raft_consensus.cc:3037] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:19.445549 28104 raft_consensus.cc:2454] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5687d30f8c674415a2e8d8ae3ee105fb in term 1.
I20260812 07:06:19.445574 28372 raft_consensus.cc:2454] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5687d30f8c674415a2e8d8ae3ee105fb in term 1.
I20260812 07:06:19.445865 28501 leader_election.cc:302] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34c9f0a8747a481fb3d34ec974ae4ad2, 5687d30f8c674415a2e8d8ae3ee105fb; no voters:
I20260812 07:06:19.446039 28754 raft_consensus.cc:2781] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:19.446259 28754 raft_consensus.cc:686] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 1 LEADER]: Becoming Leader. State: Replica: 5687d30f8c674415a2e8d8ae3ee105fb, State: Running, Role: LEADER
I20260812 07:06:19.446628 28754 consensus_queue.cc:237] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } }
I20260812 07:06:19.446872 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.442364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:36289 (local address 127.27.110.193:33085)
0812 07:06:19.442745 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.442761 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.442864 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.443116 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.443120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.443138 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.443246 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.443251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.444174 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.444178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.445430 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.445445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.446410 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.446414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.446525 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.446546 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.446622 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.446695 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":46,"thread_start_us":97,"threads_started":1}
I20260812 07:06:19.446870 28743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.442428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:33085 (local address 127.27.110.194:36289)
0812 07:06:19.442537 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.442540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.442854 (+ 314us) server_negotiation.cc:407] Connection header received
0812 07:06:19.442940 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.442943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.442984 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.443064 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.443363 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.444009 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.445595 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.446157 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.446258 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.446310 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.446659 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.446749 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:06:19.447062 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.441986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:38875 (local address 127.27.110.195:40781)
0812 07:06:19.442084 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.442088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.442106 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:06:19.442239 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.442245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.442287 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.442329 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:19.442767 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.443546 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.445299 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.445912 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.446001 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.446054 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.446878 (+ 824us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.446956 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260812 07:06:19.447636 28372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad06f80fc9d6429caa6e8361ffebde1c" candidate_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" is_pre_election: true
I20260812 07:06:19.447788 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.441664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:40781 (local address 127.27.110.193:38875)
0812 07:06:19.442001 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.442018 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:19.442120 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:19.442492 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:19.442497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:19.442517 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:19.442647 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.442654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.443663 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.443667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.445150 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:19.445163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.446198 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.446201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:19.446400 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.446678 (+ 278us) client_negotiation.cc:769] Sending connection context
0812 07:06:19.447576 (+ 898us) client_negotiation.cc:240] Negotiation successful
0812 07:06:19.447642 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":4,"thread_start_us":137,"threads_started":1}
I20260812 07:06:19.447805 28372 raft_consensus.cc:2379] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f2b6b88d5f1d41c3902e9444bf4f48c9 in current term 1: Already voted for candidate 5687d30f8c674415a2e8d8ae3ee105fb in this term.
I20260812 07:06:19.448278 28630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad06f80fc9d6429caa6e8361ffebde1c" candidate_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" is_pre_election: true
I20260812 07:06:19.448921 27970 leader_election.cc:302] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f2b6b88d5f1d41c3902e9444bf4f48c9; no voters: 34c9f0a8747a481fb3d34ec974ae4ad2, 5687d30f8c674415a2e8d8ae3ee105fb
I20260812 07:06:19.449386 28755 raft_consensus.cc:2726] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 07:06:19.450062 27812 catalog_manager.cc:5495] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5687d30f8c674415a2e8d8ae3ee105fb (127.27.110.195). New cstate: current_term: 1 leader_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5687d30f8c674415a2e8d8ae3ee105fb" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 40781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:19.475155 28744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.471549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50950 (local address 127.27.110.195:40781)
0812 07:06:19.471679 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.471682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.471696 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:06:19.471818 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.471820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.471875 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.471967 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:19.472244 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.472754 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.473625 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.473839 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.473920 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.474302 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:19.474729 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:19.474842 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.474936 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.474968 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:06:19.480921 28372 raft_consensus.cc:1261] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 1 FOLLOWER]: Refusing update from remote peer 5687d30f8c674415a2e8d8ae3ee105fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:19.480962 28104 raft_consensus.cc:1261] T ad06f80fc9d6429caa6e8361ffebde1c P f2b6b88d5f1d41c3902e9444bf4f48c9 [term 1 FOLLOWER]: Refusing update from remote peer 5687d30f8c674415a2e8d8ae3ee105fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:19.481638 28768 consensus_queue.cc:1035] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "34c9f0a8747a481fb3d34ec974ae4ad2" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:19.481832 28754 consensus_queue.cc:1035] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2b6b88d5f1d41c3902e9444bf4f48c9" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 46777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:19.491262 28778 mvcc.cc:205] Tried to move back new op lower bound from 7317579282347032576 to 7317579282216779776. Current Snapshot: MvccSnapshot[applied={T|T < 7317579282347032576}]
I20260812 07:06:19.498665 27812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:19.498844 27812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:19.501600 27812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab639bed8b6045528aa16519ccbf6678: Sending DeleteTablet for 3 replicas of tablet ad06f80fc9d6429caa6e8361ffebde1c
I20260812 07:06:19.502440 28591 tablet_service.cc:1514] Processing DeleteTablet for tablet ad06f80fc9d6429caa6e8361ffebde1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:19 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:50948
I20260812 07:06:19.502449 28058 tablet_service.cc:1514] Processing DeleteTablet for tablet ad06f80fc9d6429caa6e8361ffebde1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:19 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:55732
I20260812 07:06:19.503129 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet ad06f80fc9d6429caa6e8361ffebde1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:19 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:57020
I20260812 07:06:19.503643 28797 tablet_replica.cc:330] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: stopping tablet replica
I20260812 07:06:19.503873 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 27931
I20260812 07:06:19.512956 28797 raft_consensus.cc:2229] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:19.513355 28797 raft_consensus.cc:2258] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:19.516006 28797 ts_tablet_manager.cc:1905] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:19.519207 28797 ts_tablet_manager.cc:1918] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:19.519311 28797 log.cc:1196] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/ad06f80fc9d6429caa6e8361ffebde1c
I20260812 07:06:19.519632 28797 ts_tablet_manager.cc:1939] T ad06f80fc9d6429caa6e8361ffebde1c P 34c9f0a8747a481fb3d34ec974ae4ad2: Deleting consensus metadata
I20260812 07:06:19.520709 28794 tablet_replica.cc:330] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: stopping tablet replica
I20260812 07:06:19.521049 28794 raft_consensus.cc:2229] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:19.521025 27778 catalog_manager.cc:4841] TS 34c9f0a8747a481fb3d34ec974ae4ad2 (127.27.110.194:36289): tablet ad06f80fc9d6429caa6e8361ffebde1c (table test-table [id=1ac4c1de6cd546c6b980a71a42a8b7e4]) successfully deleted
I20260812 07:06:19.521392 28794 raft_consensus.cc:2258] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:19.522907 28794 ts_tablet_manager.cc:1905] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:19.526414 28794 ts_tablet_manager.cc:1918] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:19.526532 28794 log.cc:1196] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/ad06f80fc9d6429caa6e8361ffebde1c
I20260812 07:06:19.526897 28794 ts_tablet_manager.cc:1939] T ad06f80fc9d6429caa6e8361ffebde1c P 5687d30f8c674415a2e8d8ae3ee105fb: Deleting consensus metadata
I20260812 07:06:19.528244 27782 catalog_manager.cc:4841] TS 5687d30f8c674415a2e8d8ae3ee105fb (127.27.110.195:40781): tablet ad06f80fc9d6429caa6e8361ffebde1c (table test-table [id=1ac4c1de6cd546c6b980a71a42a8b7e4]) successfully deleted
W20260812 07:06:19.529173 27783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:46777 (error 108)
I20260812 07:06:19.529404 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28214
I20260812 07:06:19.538444 28736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.538075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:46777 (local address 127.0.0.1:55736)
0812 07:06:19.538268 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:19.538348 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:46777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260812 07:06:19.538657 27783 catalog_manager.cc:4584] TS f2b6b88d5f1d41c3902e9444bf4f48c9 (127.27.110.193:46777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad06f80fc9d6429caa6e8361ffebde1c: Network error: Client connection negotiation failed: client connection to 127.27.110.193:46777: connect: Connection refused (error 111)
I20260812 07:06:19.544147 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28471
I20260812 07:06:19.559342 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 27747
2026-08-12T07:06:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2697 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T07:06:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:19Z Disabled control of system clock
I20260812 07:06:19.600940 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:38673
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:38673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:06:19.712363 28803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:19.712677 28803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:19.712716 28803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:19.716969 28803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:19.717036 28803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:19.717051 28803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:19.717067 28803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:19.717082 28803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:19.721964 28803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:38673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:19.723208 28803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:19.724660 28803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:19.733402 28821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:19.733402 28810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:19.733755 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:19.733819 28803 server_base.cc:1034] running on GCE node
I20260812 07:06:19.734418 28803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:19.735632 28803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:19.736807 28803 hybrid_clock.cc:648] HybridClock initialized: now 1786518379736776 us; error 44 us; skew 500 ppm
I20260812 07:06:19.738730 28803 webserver.cc:460] Webserver started at http://127.27.110.254:42169/ using document root <none> and password file <none>
I20260812 07:06:19.739338 28803 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:19.739404 28803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:19.739580 28803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:19.741704 28803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "efd167cd4de1427eae693e1390301926"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "60ce7576556e68a50b19d4415a8a5096"
server_key_iv: "6a1bd7f7f1759a666a9315e309362977"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.742211 28803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "efd167cd4de1427eae693e1390301926"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "60ce7576556e68a50b19d4415a8a5096"
server_key_iv: "6a1bd7f7f1759a666a9315e309362977"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.745921 28803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:06:19.748486 28829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.749768 28803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:19.749867 28803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "efd167cd4de1427eae693e1390301926"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "60ce7576556e68a50b19d4415a8a5096"
server_key_iv: "6a1bd7f7f1759a666a9315e309362977"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.749950 28803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:19.762710 28803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:19.763417 28803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:19.763568 28803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:19.770650 28803 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:38673
I20260812 07:06:19.770666 28937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:38673 every 8 connection(s)
I20260812 07:06:19.771682 28803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:19.774884 28939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:19.777210 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28803
I20260812 07:06:19.777349 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:19.777647 28091 external_mini_cluster.cc:1428] Setting key 4ae45f5c7f44428f2133fe6b70a07abc
I20260812 07:06:19.781380 28939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926: Bootstrap starting.
I20260812 07:06:19.784078 28939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:19.784965 28939 log.cc:824] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:19.787233 28939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926: No bootstrap required, opened a new log
I20260812 07:06:19.788717 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:19.779231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.27.110.254:38673)
0812 07:06:19.779904 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:19.779928 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:19.779977 (+ 49us) server_negotiation.cc:407] Connection header received
0812 07:06:19.780814 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:19.780838 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:19.781340 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:19.781706 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:19.782170 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.783654 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.784669 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:19.785473 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:19.785599 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:19.786629 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:19.786634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:19.786650 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:19.787202 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:19.787306 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:19.787338 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:19.787546 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:19.787661 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 07:06:19.787968 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260812 07:06:19.790808 28939 raft_consensus.cc:348] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd167cd4de1427eae693e1390301926" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 38673 } }
I20260812 07:06:19.791118 28939 raft_consensus.cc:374] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:19.791172 28939 raft_consensus.cc:729] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efd167cd4de1427eae693e1390301926, State: Initialized, Role: FOLLOWER
I20260812 07:06:19.791652 28939 consensus_queue.cc:260] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd167cd4de1427eae693e1390301926" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 38673 } }
I20260812 07:06:19.791824 28939 raft_consensus.cc:388] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:19.791880 28939 raft_consensus.cc:482] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:19.791954 28939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:19.792851 28939 raft_consensus.cc:504] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd167cd4de1427eae693e1390301926" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 38673 } }
I20260812 07:06:19.793144 28939 leader_election.cc:302] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efd167cd4de1427eae693e1390301926; no voters:
I20260812 07:06:19.793401 28939 leader_election.cc:288] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:19.793558 28945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:19.793836 28945 raft_consensus.cc:686] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [term 1 LEADER]: Becoming Leader. State: Replica: efd167cd4de1427eae693e1390301926, State: Running, Role: LEADER
I20260812 07:06:19.794173 28945 consensus_queue.cc:237] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd167cd4de1427eae693e1390301926" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 38673 } }
I20260812 07:06:19.794683 28939 sys_catalog.cc:564] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:19.795915 28949 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efd167cd4de1427eae693e1390301926. Latest consensus state: current_term: 1 leader_uuid: "efd167cd4de1427eae693e1390301926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd167cd4de1427eae693e1390301926" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 38673 } } }
I20260812 07:06:19.796057 28949 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:19.796329 28948 sys_catalog.cc:455] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efd167cd4de1427eae693e1390301926" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efd167cd4de1427eae693e1390301926" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 38673 } } }
I20260812 07:06:19.796422 28948 sys_catalog.cc:458] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:19.796851 28958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:19.799598 28958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:19.804523 28958 catalog_manager.cc:1347] Generated new cluster ID: 305c766f2cb44d49bc4240fe0b344ca5
I20260812 07:06:19.804586 28958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:19.811888 28958 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:19.812742 28958 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:19.823357 28958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926: Generated new TSK 0
I20260812 07:06:19.824159 28958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:19.830101 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:38673
--builtin_ntp_servers=127.27.110.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:06:19.953312 28979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:19.953560 28979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:19.953598 28979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:19.957769 28979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:19.957823 28979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:19.957901 28979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:06:19.962678 28979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:38673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:19.963785 28979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:19.965195 28979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:19.975118 28994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:19.975179 28993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:19.975258 28979 server_base.cc:1034] running on GCE node
W20260812 07:06:19.975152 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:19.975741 28979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:19.976382 28979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:19.977589 28979 hybrid_clock.cc:648] HybridClock initialized: now 1786518379977559 us; error 40 us; skew 500 ppm
I20260812 07:06:19.979504 28979 webserver.cc:460] Webserver started at http://127.27.110.193:46503/ using document root <none> and password file <none>
I20260812 07:06:19.980034 28979 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:19.980093 28979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:19.980259 28979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:19.982295 28979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "6a14f5b520024041ac8418b24aa23a37"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "59431df461d442ad40d092ec4d78241e"
server_key_iv: "7824312046211ab28b6d079292f8d329"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.982798 28979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "6a14f5b520024041ac8418b24aa23a37"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "59431df461d442ad40d092ec4d78241e"
server_key_iv: "7824312046211ab28b6d079292f8d329"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.986356 28979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:06:19.988855 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:19.990137 28979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:19.990243 28979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "6a14f5b520024041ac8418b24aa23a37"
format_stamp: "Formatted at 2026-08-12 07:06:19 on dist-test-slave-w4v5"
server_key: "59431df461d442ad40d092ec4d78241e"
server_key_iv: "7824312046211ab28b6d079292f8d329"
server_key_version: "encryptionkey@0"
I20260812 07:06:19.990332 28979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:20.005301 28979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:20.005982 28979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:20.006142 28979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:20.006700 28979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:20.007761 28979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:20.007823 28979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.007864 28979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:20.007880 28979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.017413 28979 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:40593
I20260812 07:06:20.017438 29230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:40593 every 8 connection(s)
I20260812 07:06:20.018443 28979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:20.028118 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.020414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:60675 (local address 127.27.110.254:38673)
0812 07:06:20.020595 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.020600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.021747 (+ 1147us) server_negotiation.cc:407] Connection header received
0812 07:06:20.023153 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.023157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.023210 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.023289 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:20.024271 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.024845 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.026037 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.026269 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.026352 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.027545 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:20.027548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:20.027550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:20.027564 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:20.027585 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:20.027592 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:20.027655 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.027870 (+ 215us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.027938 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:06:20.028716 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 28979
I20260812 07:06:20.028861 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:20.028992 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.020831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:38673 (local address 127.27.110.193:60675)
0812 07:06:20.021557 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.021601 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.022881 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.023477 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.023484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.023604 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:20.024105 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.024119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.025015 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.025018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.025911 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.025922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.026412 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.026415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.026556 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.027271 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:20.027273 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:20.027314 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:20.027379 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:20.027686 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:20.027695 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:20.027701 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.027807 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.028037 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":122,"threads_started":1}
I20260812 07:06:20.029215 28091 external_mini_cluster.cc:1428] Setting key 736937de4bfe68876afab8c667520e34
I20260812 07:06:20.030323 29231 heartbeater.cc:347] Connected to a master server at 127.27.110.254:38673
I20260812 07:06:20.030589 29231 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:20.031226 29231 heartbeater.cc:511] Master 127.27.110.254:38673 requested a full tablet report, sending...
I20260812 07:06:20.031520 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:38673
--builtin_ntp_servers=127.27.110.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:06:20.032728 28864 ts_manager.cc:194] Registered new tserver with Master: 6a14f5b520024041ac8418b24aa23a37 (127.27.110.193:40593)
I20260812 07:06:20.034323 28864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.193:60675
W20260812 07:06:20.148420 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:20.148733 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:20.148804 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:20.152792 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:20.152878 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:20.152969 29237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:06:20.157888 29237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:38673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:20.159049 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:20.160488 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:20.170037 29249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:20.170054 29250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:20.170058 29253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:20.170403 29237 server_base.cc:1034] running on GCE node
I20260812 07:06:20.170815 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:20.171579 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:20.172776 29237 hybrid_clock.cc:648] HybridClock initialized: now 1786518380172744 us; error 45 us; skew 500 ppm
I20260812 07:06:20.174778 29237 webserver.cc:460] Webserver started at http://127.27.110.194:37089/ using document root <none> and password file <none>
I20260812 07:06:20.175429 29237 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:20.175498 29237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:20.175748 29237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:20.177866 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "809d7c85ec734ecaa9ffc20fea340ced"
format_stamp: "Formatted at 2026-08-12 07:06:20 on dist-test-slave-w4v5"
server_key: "ad3c5d183913d17f57ce6a4173d67571"
server_key_iv: "147ab6f538a420c0675c192833c09b8e"
server_key_version: "encryptionkey@0"
I20260812 07:06:20.178448 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "809d7c85ec734ecaa9ffc20fea340ced"
format_stamp: "Formatted at 2026-08-12 07:06:20 on dist-test-slave-w4v5"
server_key: "ad3c5d183913d17f57ce6a4173d67571"
server_key_iv: "147ab6f538a420c0675c192833c09b8e"
server_key_version: "encryptionkey@0"
I20260812 07:06:20.182309 29237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260812 07:06:20.184875 29261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.186071 29237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:20.186170 29237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "809d7c85ec734ecaa9ffc20fea340ced"
format_stamp: "Formatted at 2026-08-12 07:06:20 on dist-test-slave-w4v5"
server_key: "ad3c5d183913d17f57ce6a4173d67571"
server_key_iv: "147ab6f538a420c0675c192833c09b8e"
server_key_version: "encryptionkey@0"
I20260812 07:06:20.186308 29237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:20.212399 29237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:20.213370 29237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:20.213590 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:20.214416 29237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:20.219836 29237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:20.219892 29237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.219956 29237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:20.219990 29237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.236886 29237 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:43603
I20260812 07:06:20.236905 29488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:43603 every 8 connection(s)
I20260812 07:06:20.238212 29237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:20.242622 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29237
I20260812 07:06:20.242781 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:20.243165 28091 external_mini_cluster.cc:1428] Setting key 871677321339fb557de4406b59fc5f5b
I20260812 07:06:20.249184 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:38673
--builtin_ntp_servers=127.27.110.212:38805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:06:20.252871 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.242653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:37509 (local address 127.27.110.254:38673)
0812 07:06:20.242811 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.242818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.242838 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:06:20.244582 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.244587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.244665 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.244778 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:20.245721 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.246690 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.248866 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.249130 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.249258 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.252228 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:20.252234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:20.252241 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:20.252258 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:20.252284 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:20.252297 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:20.252389 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.252568 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.252651 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:06:20.254144 29493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.241358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:38673 (local address 127.27.110.194:37509)
0812 07:06:20.242395 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.242460 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.244345 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.244938 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.244945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.245034 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:20.245572 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.245584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.246865 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.246868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.248689 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.248702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.249537 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.249543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.249818 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.251896 (+ 2078us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:20.251898 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:20.251962 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:20.252040 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:20.252397 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:20.252405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:20.252411 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.252514 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.252811 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":153,"threads_started":1}
I20260812 07:06:20.255771 29490 heartbeater.cc:347] Connected to a master server at 127.27.110.254:38673
I20260812 07:06:20.256076 29490 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:20.256850 29490 heartbeater.cc:511] Master 127.27.110.254:38673 requested a full tablet report, sending...
I20260812 07:06:20.258154 28864 ts_manager.cc:194] Registered new tserver with Master: 809d7c85ec734ecaa9ffc20fea340ced (127.27.110.194:43603)
I20260812 07:06:20.259052 28864 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.194:37509
W20260812 07:06:20.373176 29494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:20.373440 29494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:20.373519 29494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:20.377573 29494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:20.377650 29494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:20.377734 29494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:06:20.382464 29494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:38805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:38673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:20.383579 29494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:20.384922 29494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:20.394840 29503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:20.394841 29504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:20.395200 29494 server_base.cc:1034] running on GCE node
W20260812 07:06:20.394840 29507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:20.395792 29494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:20.396515 29494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:20.397639 29494 hybrid_clock.cc:648] HybridClock initialized: now 1786518380397617 us; error 40 us; skew 500 ppm
I20260812 07:06:20.399560 29494 webserver.cc:460] Webserver started at http://127.27.110.195:39685/ using document root <none> and password file <none>
I20260812 07:06:20.400184 29494 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:20.400288 29494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:20.400544 29494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:20.402704 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "ad6d3a16f3dc43c4819e089f249c80fc"
format_stamp: "Formatted at 2026-08-12 07:06:20 on dist-test-slave-w4v5"
server_key: "91741df9613012fc272363861e35ecc7"
server_key_iv: "bbadee6f62f50859f9f22bcd45bd449b"
server_key_version: "encryptionkey@0"
I20260812 07:06:20.403360 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "ad6d3a16f3dc43c4819e089f249c80fc"
format_stamp: "Formatted at 2026-08-12 07:06:20 on dist-test-slave-w4v5"
server_key: "91741df9613012fc272363861e35ecc7"
server_key_iv: "bbadee6f62f50859f9f22bcd45bd449b"
server_key_version: "encryptionkey@0"
I20260812 07:06:20.407217 29494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:06:20.409731 29520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.410959 29494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:20.411149 29494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "ad6d3a16f3dc43c4819e089f249c80fc"
format_stamp: "Formatted at 2026-08-12 07:06:20 on dist-test-slave-w4v5"
server_key: "91741df9613012fc272363861e35ecc7"
server_key_iv: "bbadee6f62f50859f9f22bcd45bd449b"
server_key_version: "encryptionkey@0"
I20260812 07:06:20.411266 29494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:20.437160 29494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:20.437955 29494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:20.438180 29494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:20.438844 29494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:20.439895 29494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:20.439966 29494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.440037 29494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:20.440090 29494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:20.449381 29494 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44933
I20260812 07:06:20.449430 29763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44933 every 8 connection(s)
I20260812 07:06:20.450335 29494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:20.459497 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29494
I20260812 07:06:20.459625 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:20.459770 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.452240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:47203 (local address 127.27.110.254:38673)
0812 07:06:20.452392 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.452398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.453587 (+ 1189us) server_negotiation.cc:407] Connection header received
0812 07:06:20.454990 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.454993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.455048 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.455176 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:20.456048 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.456570 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.457673 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.457889 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.457970 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.459028 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:20.459031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:20.459033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:20.459050 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:20.459130 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:20.459146 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:20.459215 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.459459 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.459522 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:06:20.459936 28091 external_mini_cluster.cc:1428] Setting key bb5e37d34b1a38d60d0949ac341fc6ed
I20260812 07:06:20.460733 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.452664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:38673 (local address 127.27.110.195:47203)
0812 07:06:20.453440 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.453481 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.454797 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.455319 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.455326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.455407 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:20.455900 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.455912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.456681 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.456684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.457547 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.457555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.458022 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.458025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.458164 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.458722 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:20.458724 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:20.458762 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:20.458824 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:20.459256 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:20.459268 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:20.459275 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.459416 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.459743 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":160,"threads_started":1}
I20260812 07:06:20.461795 29765 heartbeater.cc:347] Connected to a master server at 127.27.110.254:38673
I20260812 07:06:20.462060 29765 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:20.462592 29765 heartbeater.cc:511] Master 127.27.110.254:38673 requested a full tablet report, sending...
I20260812 07:06:20.463608 28863 ts_manager.cc:194] Registered new tserver with Master: ad6d3a16f3dc43c4819e089f249c80fc (127.27.110.195:44933)
I20260812 07:06:20.464147 28863 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.195:47203
I20260812 07:06:20.473820 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:20.479550 28942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.475882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.27.110.254:38673)
0812 07:06:20.476033 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.476038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.476238 (+ 200us) server_negotiation.cc:407] Connection header received
0812 07:06:20.476316 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.476318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.476356 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.476420 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:20.476680 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.477171 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.477963 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.478165 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.478239 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.479012 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:20.479015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:20.479017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:20.479028 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:20.479046 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:20.479054 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:20.479228 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.479327 (+ 99us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.479380 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:06:20.482460 28863 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:20.485510 28863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:20.501385 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.494213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44933 (local address 127.0.0.1:39774)
0812 07:06:20.495366 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.495417 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.495546 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.496819 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.496823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.496842 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:20.496933 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.496942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.498280 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.498283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.500050 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.500057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.500825 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.500828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.500935 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.500971 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.501091 (+ 120us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.501225 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":502,"thread_start_us":73,"threads_started":1}
I20260812 07:06:20.501389 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.494367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.27.110.195:44933)
0812 07:06:20.496250 (+ 1883us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.496261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.496283 (+ 22us) server_negotiation.cc:407] Connection header received
0812 07:06:20.496356 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.496360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.496527 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.496679 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:20.497058 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.498166 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.500162 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.500687 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.500826 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.501006 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.501124 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.501225 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":139,"threads_started":1}
I20260812 07:06:20.501919 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.494273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:43603 (local address 127.0.0.1:52040)
0812 07:06:20.495641 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.495656 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.495764 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.496577 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.496583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.496608 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:20.496744 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.496754 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.498836 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.498838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.499764 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.499773 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.501074 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.501078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.501244 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.501276 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.501353 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.501429 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"mutex_wait_us":78,"thread_start_us":108,"threads_started":1}
I20260812 07:06:20.502276 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.494991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40593 (local address 127.0.0.1:52596)
0812 07:06:20.495901 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.495915 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.495991 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.496643 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.496646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.496665 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:20.496761 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.496767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.498258 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.498261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.499853 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.499865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.501074 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.501078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.501965 (+ 887us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.501993 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.502071 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.502149 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":99,"thread_start_us":130,"threads_started":1}
I20260812 07:06:20.502342 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.494698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.27.110.194:43603)
0812 07:06:20.495061 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.495140 (+ 79us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.495832 (+ 692us) server_negotiation.cc:407] Connection header received
0812 07:06:20.495965 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.495971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.496238 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.496401 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:20.496895 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.498716 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.499889 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.500710 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.500833 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.501006 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.502092 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.502189 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":124,"threads_started":1}
I20260812 07:06:20.502748 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.495244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.27.110.193:40593)
0812 07:06:20.496003 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.496011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.496037 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:06:20.496131 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.496137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.496332 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.496481 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:20.496894 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.498112 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.499958 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.500709 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.500833 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.501006 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.502539 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.502625 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":127,"threads_started":1}
I20260812 07:06:20.504508 29625 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdd67c999764749bf2d14810a59d800 (DEFAULT_TABLE table=test-table [id=7e6eda58b7c94aaebe97174e24819c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:20.505236 29355 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdd67c999764749bf2d14810a59d800 (DEFAULT_TABLE table=test-table [id=7e6eda58b7c94aaebe97174e24819c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:20.505666 29625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdd67c999764749bf2d14810a59d800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:20.505511 29094 tablet_service.cc:1467] Processing CreateTablet for tablet 5cdd67c999764749bf2d14810a59d800 (DEFAULT_TABLE table=test-table [id=7e6eda58b7c94aaebe97174e24819c14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:20.506305 29355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdd67c999764749bf2d14810a59d800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:20.506541 29094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cdd67c999764749bf2d14810a59d800. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:20.512404 29795 tablet_bootstrap.cc:492] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Bootstrap starting.
I20260812 07:06:20.514150 29798 tablet_bootstrap.cc:492] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Bootstrap starting.
I20260812 07:06:20.514930 29795 tablet_bootstrap.cc:654] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:20.515868 29795 log.cc:824] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:20.516225 29798 tablet_bootstrap.cc:654] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:20.517122 29798 log.cc:824] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:20.517189 29796 tablet_bootstrap.cc:492] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37: Bootstrap starting.
I20260812 07:06:20.518106 29795 tablet_bootstrap.cc:492] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: No bootstrap required, opened a new log
I20260812 07:06:20.518301 29795 ts_tablet_manager.cc:1397] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:06:20.519222 29796 tablet_bootstrap.cc:654] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:20.519266 29798 tablet_bootstrap.cc:492] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: No bootstrap required, opened a new log
I20260812 07:06:20.519433 29798 ts_tablet_manager.cc:1397] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:20.520015 29796 log.cc:824] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:20.521638 29796 tablet_bootstrap.cc:492] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37: No bootstrap required, opened a new log
I20260812 07:06:20.521839 29796 ts_tablet_manager.cc:1397] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260812 07:06:20.522802 29795 raft_consensus.cc:348] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.523113 29795 raft_consensus.cc:374] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:20.523171 29795 raft_consensus.cc:729] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6d3a16f3dc43c4819e089f249c80fc, State: Initialized, Role: FOLLOWER
I20260812 07:06:20.523624 29795 consensus_queue.cc:260] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.523553 29798 raft_consensus.cc:348] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.523833 29798 raft_consensus.cc:374] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:20.523890 29798 raft_consensus.cc:729] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 809d7c85ec734ecaa9ffc20fea340ced, State: Initialized, Role: FOLLOWER
I20260812 07:06:20.524312 29798 consensus_queue.cc:260] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.524672 29795 ts_tablet_manager.cc:1428] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260812 07:06:20.525013 29490 heartbeater.cc:503] Master 127.27.110.254:38673 was elected leader, sending a full tablet report...
I20260812 07:06:20.525008 29765 heartbeater.cc:503] Master 127.27.110.254:38673 was elected leader, sending a full tablet report...
I20260812 07:06:20.526206 29798 ts_tablet_manager.cc:1428] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:06:20.527129 29796 raft_consensus.cc:348] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.527406 29796 raft_consensus.cc:374] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:20.527468 29796 raft_consensus.cc:729] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a14f5b520024041ac8418b24aa23a37, State: Initialized, Role: FOLLOWER
I20260812 07:06:20.527902 29796 consensus_queue.cc:260] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.528594 29231 heartbeater.cc:503] Master 127.27.110.254:38673 was elected leader, sending a full tablet report...
I20260812 07:06:20.528815 29796 ts_tablet_manager.cc:1428] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260812 07:06:20.672266 29803 raft_consensus.cc:482] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:20.672509 29803 raft_consensus.cc:504] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.673750 29803 leader_election.cc:288] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 809d7c85ec734ecaa9ffc20fea340ced (127.27.110.194:43603), 6a14f5b520024041ac8418b24aa23a37 (127.27.110.193:40593)
I20260812 07:06:20.679400 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.674278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:46459 (local address 127.27.110.193:40593)
0812 07:06:20.674409 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.674414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.674433 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:06:20.674583 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.674588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.674673 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.674762 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:20.675118 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.675938 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.677932 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.678574 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.678690 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.678740 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.679178 (+ 438us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.679264 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:06:20.679400 29791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.674170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:51585 (local address 127.27.110.194:43603)
0812 07:06:20.674302 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.674306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.674672 (+ 366us) server_negotiation.cc:407] Connection header received
0812 07:06:20.674743 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.674749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.674804 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.674888 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:20.675198 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.675978 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.677932 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.678559 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.678690 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.678739 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.679178 (+ 439us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.679264 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:06:20.679415 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.674064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40593 (local address 127.27.110.195:46459)
0812 07:06:20.674213 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.674231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.674362 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.674791 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.674794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.674808 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:20.674955 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.674963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.676103 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.676107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.677775 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.677785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.678717 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.678721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.678953 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.679014 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.679129 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.679239 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":291}
I20260812 07:06:20.679425 29806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.674064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:43603 (local address 127.27.110.195:51585)
0812 07:06:20.674578 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.674589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:20.674666 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:20.674901 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:20.674905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:20.674924 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:20.675102 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.675108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.676103 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.676107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.677772 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:20.677785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.678717 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.678721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:20.678947 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.679015 (+ 68us) client_negotiation.cc:769] Sending connection context
0812 07:06:20.679129 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 07:06:20.679239 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":435,"thread_start_us":117,"threads_started":1}
I20260812 07:06:20.680222 29131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd67c999764749bf2d14810a59d800" candidate_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a14f5b520024041ac8418b24aa23a37" is_pre_election: true
I20260812 07:06:20.680220 29412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd67c999764749bf2d14810a59d800" candidate_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d7c85ec734ecaa9ffc20fea340ced" is_pre_election: true
I20260812 07:06:20.680521 29131 raft_consensus.cc:2454] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6d3a16f3dc43c4819e089f249c80fc in term 0.
I20260812 07:06:20.680536 29412 raft_consensus.cc:2454] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6d3a16f3dc43c4819e089f249c80fc in term 0.
I20260812 07:06:20.680994 29522 leader_election.cc:302] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a14f5b520024041ac8418b24aa23a37, ad6d3a16f3dc43c4819e089f249c80fc; no voters:
I20260812 07:06:20.681529 29803 raft_consensus.cc:2781] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:20.681630 29803 raft_consensus.cc:482] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:20.681694 29803 raft_consensus.cc:3037] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:20.682837 29803 raft_consensus.cc:504] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.683280 29803 leader_election.cc:288] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [CANDIDATE]: Term 1 election: Requested vote from peers 809d7c85ec734ecaa9ffc20fea340ced (127.27.110.194:43603), 6a14f5b520024041ac8418b24aa23a37 (127.27.110.193:40593)
I20260812 07:06:20.683566 29412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd67c999764749bf2d14810a59d800" candidate_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "809d7c85ec734ecaa9ffc20fea340ced"
I20260812 07:06:20.683638 29131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cdd67c999764749bf2d14810a59d800" candidate_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a14f5b520024041ac8418b24aa23a37"
I20260812 07:06:20.683692 29412 raft_consensus.cc:3037] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:20.683784 29131 raft_consensus.cc:3037] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:20.684800 29131 raft_consensus.cc:2454] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6d3a16f3dc43c4819e089f249c80fc in term 1.
I20260812 07:06:20.685009 29412 raft_consensus.cc:2454] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6d3a16f3dc43c4819e089f249c80fc in term 1.
I20260812 07:06:20.685169 29522 leader_election.cc:302] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a14f5b520024041ac8418b24aa23a37, ad6d3a16f3dc43c4819e089f249c80fc; no voters:
I20260812 07:06:20.685345 29803 raft_consensus.cc:2781] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:20.685612 29803 raft_consensus.cc:686] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 1 LEADER]: Becoming Leader. State: Replica: ad6d3a16f3dc43c4819e089f249c80fc, State: Running, Role: LEADER
I20260812 07:06:20.685967 29803 consensus_queue.cc:237] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } }
I20260812 07:06:20.689034 28864 catalog_manager.cc:5495] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6d3a16f3dc43c4819e089f249c80fc (127.27.110.195). New cstate: current_term: 1 leader_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6d3a16f3dc43c4819e089f249c80fc" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 } health_report { overall_health: UNKNOWN } } }
W20260812 07:06:20.701484 29766 tablet.cc:2369] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:20.737248 29788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.733375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.27.110.195:44933)
0812 07:06:20.733526 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:20.733531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:20.733547 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:06:20.733694 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:20.733696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:20.733757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:20.733882 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:20.734161 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.734694 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.735692 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:20.735915 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:20.736029 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:20.736364 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:20.736792 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:20.736960 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:20.737057 (+ 97us) server_negotiation.cc:299] Negotiation successful
0812 07:06:20.737093 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 07:06:20.739614 29492 tablet.cc:2369] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:20.743579 29131 raft_consensus.cc:1261] T 5cdd67c999764749bf2d14810a59d800 P 6a14f5b520024041ac8418b24aa23a37 [term 1 FOLLOWER]: Refusing update from remote peer ad6d3a16f3dc43c4819e089f249c80fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:20.743774 29412 raft_consensus.cc:1261] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 1 FOLLOWER]: Refusing update from remote peer ad6d3a16f3dc43c4819e089f249c80fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:20.744354 29803 consensus_queue.cc:1035] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a14f5b520024041ac8418b24aa23a37" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:20.744549 29810 consensus_queue.cc:1035] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "809d7c85ec734ecaa9ffc20fea340ced" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 43603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:20.752991 29814 mvcc.cc:205] Tried to move back new op lower bound from 7317579287517364224 to 7317579287292137472. Current Snapshot: MvccSnapshot[applied={T|T < 7317579287517364224}]
I20260812 07:06:20.755002 29815 mvcc.cc:205] Tried to move back new op lower bound from 7317579287517364224 to 7317579287292137472. Current Snapshot: MvccSnapshot[applied={T|T < 7317579287517364224}]
I20260812 07:06:20.761818 28863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:20.762025 28863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:20.765014 28863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efd167cd4de1427eae693e1390301926: Sending DeleteTablet for 3 replicas of tablet 5cdd67c999764749bf2d14810a59d800
I20260812 07:06:20.765750 29355 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdd67c999764749bf2d14810a59d800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:20 UTC) from {username='slave'} at 127.0.0.1:52040
I20260812 07:06:20.765852 29625 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdd67c999764749bf2d14810a59d800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:20 UTC) from {username='slave'} at 127.0.0.1:39774
I20260812 07:06:20.766220 29823 tablet_replica.cc:330] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: stopping tablet replica
I20260812 07:06:20.766448 29094 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cdd67c999764749bf2d14810a59d800 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:20 UTC) from {username='slave'} at 127.0.0.1:52596
I20260812 07:06:20.766659 29823 raft_consensus.cc:2229] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:20.766914 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28979
I20260812 07:06:20.782194 29823 raft_consensus.cc:2258] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:20.785986 29823 ts_tablet_manager.cc:1905] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:20.789505 29823 ts_tablet_manager.cc:1918] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:20.789727 29823 log.cc:1196] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/5cdd67c999764749bf2d14810a59d800
I20260812 07:06:20.790134 29823 ts_tablet_manager.cc:1939] T 5cdd67c999764749bf2d14810a59d800 P 809d7c85ec734ecaa9ffc20fea340ced: Deleting consensus metadata
I20260812 07:06:20.791981 28832 catalog_manager.cc:4841] TS 809d7c85ec734ecaa9ffc20fea340ced (127.27.110.194:43603): tablet 5cdd67c999764749bf2d14810a59d800 (table test-table [id=7e6eda58b7c94aaebe97174e24819c14]) successfully deleted
I20260812 07:06:20.803364 29830 tablet_replica.cc:330] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: stopping tablet replica
I20260812 07:06:20.803907 29830 raft_consensus.cc:2229] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:20.804493 29830 raft_consensus.cc:2258] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:20.806348 29830 ts_tablet_manager.cc:1905] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:20.810108 29830 ts_tablet_manager.cc:1918] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:20.810230 29830 log.cc:1196] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/5cdd67c999764749bf2d14810a59d800
I20260812 07:06:20.810674 29830 ts_tablet_manager.cc:1939] T 5cdd67c999764749bf2d14810a59d800 P ad6d3a16f3dc43c4819e089f249c80fc: Deleting consensus metadata
I20260812 07:06:20.812068 28836 catalog_manager.cc:4841] TS ad6d3a16f3dc43c4819e089f249c80fc (127.27.110.195:44933): tablet 5cdd67c999764749bf2d14810a59d800 (table test-table [id=7e6eda58b7c94aaebe97174e24819c14]) successfully deleted
W20260812 07:06:20.813333 28830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:40593 (error 108)
I20260812 07:06:20.814436 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:20.814109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40593 (local address 127.0.0.1:52608)
0812 07:06:20.814250 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:20.814331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 07:06:20.814569 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29237
W20260812 07:06:20.814653 28830 catalog_manager.cc:4584] TS 6a14f5b520024041ac8418b24aa23a37 (127.27.110.193:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cdd67c999764749bf2d14810a59d800: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40593: connect: Connection refused (error 111)
I20260812 07:06:20.848207 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29494
I20260812 07:06:20.860514 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 28803
2026-08-12T07:06:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1297 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:20 dist-test-slave-w4v5 krb5kdc[29839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:20 dist-test-slave-w4v5 krb5kdc[29839](info): set up 2 sockets
Aug 12 07:06:20 dist-test-slave-w4v5 krb5kdc[29839](info): commencing operation
krb5kdc: starting...
W20260812 07:06:22.246904 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.358s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:22 dist-test-slave-w4v5 krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:22.376468 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39299
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:22.488554 29873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:22.488850 29873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:22.488924 29873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:22.492982 29873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:22.493067 29873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:22.493106 29873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:22.493173 29873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:22.493227 29873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:22.498324 29873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:22.499451 29873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:22.500520 29873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:22.509742 29880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:22.509742 29882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:22.509858 29887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:22.509967 29873 server_base.cc:1034] running on GCE node
I20260812 07:06:22.510658 29873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:22.511811 29873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:22.513005 29873 hybrid_clock.cc:648] HybridClock initialized: now 1786518382512988 us; error 27 us; skew 500 ppm
Aug 12 07:06:22 dist-test-slave-w4v5 krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518382, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:22.515861 29873 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:22.516714 29873 webserver.cc:460] Webserver started at http://127.27.110.254:33795/ using document root <none> and password file <none>
I20260812 07:06:22.517221 29873 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:22.517279 29873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:22.517441 29873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:22.519438 29873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "1360bddbac634f32b00cb0f64b371614"
format_stamp: "Formatted at 2026-08-12 07:06:22 on dist-test-slave-w4v5"
server_key: "bd255b7f63c00b410e9c43cc15f08248"
server_key_iv: "7447f509fdcec310cab486df3ad4fc20"
server_key_version: "encryptionkey@0"
I20260812 07:06:22.519943 29873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "1360bddbac634f32b00cb0f64b371614"
format_stamp: "Formatted at 2026-08-12 07:06:22 on dist-test-slave-w4v5"
server_key: "bd255b7f63c00b410e9c43cc15f08248"
server_key_iv: "7447f509fdcec310cab486df3ad4fc20"
server_key_version: "encryptionkey@0"
I20260812 07:06:22.523584 29873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 07:06:22.526392 29907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:22.527824 29873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:22.527931 29873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "1360bddbac634f32b00cb0f64b371614"
format_stamp: "Formatted at 2026-08-12 07:06:22 on dist-test-slave-w4v5"
server_key: "bd255b7f63c00b410e9c43cc15f08248"
server_key_iv: "7447f509fdcec310cab486df3ad4fc20"
server_key_version: "encryptionkey@0"
I20260812 07:06:22.528024 29873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:22.551857 29873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:22.555192 29873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:22.555348 29873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:22.562637 29873 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39299
I20260812 07:06:22.562745 30013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39299 every 8 connection(s)
I20260812 07:06:22.563678 29873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:22.566498 30014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:22.572014 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: Bootstrap starting.
I20260812 07:06:22.573140 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 29873
I20260812 07:06:22.573303 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:22.573627 28091 external_mini_cluster.cc:1428] Setting key 970f715549ea216b24b669e63fdaa862
I20260812 07:06:22.575033 30014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:22.575990 30014 log.cc:824] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:22.578362 30014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: No bootstrap required, opened a new log
I20260812 07:06:22.582368 30014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1360bddbac634f32b00cb0f64b371614" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39299 } }
I20260812 07:06:22.582587 30014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:22.582664 30014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1360bddbac634f32b00cb0f64b371614, State: Initialized, Role: FOLLOWER
I20260812 07:06:22.583240 30014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1360bddbac634f32b00cb0f64b371614" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39299 } }
Aug 12 07:06:22 dist-test-slave-w4v5 krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:22.583429 30014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:22.583500 30014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:22.583609 30014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:22.584903 30014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1360bddbac634f32b00cb0f64b371614" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39299 } }
I20260812 07:06:22.585311 30014 leader_election.cc:302] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1360bddbac634f32b00cb0f64b371614; no voters:
I20260812 07:06:22.585728 30014 leader_election.cc:288] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:22.585870 30025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:22.586138 30025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [term 1 LEADER]: Becoming Leader. State: Replica: 1360bddbac634f32b00cb0f64b371614, State: Running, Role: LEADER
I20260812 07:06:22.586493 30025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1360bddbac634f32b00cb0f64b371614" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39299 } }
I20260812 07:06:22.587129 30014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:22.588346 30027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1360bddbac634f32b00cb0f64b371614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1360bddbac634f32b00cb0f64b371614" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39299 } } }
I20260812 07:06:22.588478 30027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:22.588792 30026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1360bddbac634f32b00cb0f64b371614. Latest consensus state: current_term: 1 leader_uuid: "1360bddbac634f32b00cb0f64b371614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1360bddbac634f32b00cb0f64b371614" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39299 } } }
I20260812 07:06:22.588905 30026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:22.590751 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:22.574914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.27.110.254:39299)
0812 07:06:22.575366 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:22.575387 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:22.575430 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:06:22.576319 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:22.576348 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:22.576699 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:22.577084 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:22.577784 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:22.579301 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:22.580232 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:22.580971 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:22.581185 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:22.584161 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:22.584168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:22.584185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:22.584217 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:22.586326 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:22.586958 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:22.586964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:22.586970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:22.587192 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:22.587515 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:22.587519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:22.587520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:22.588081 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:22.588270 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:22.589242 (+ 972us) server_negotiation.cc:299] Negotiation successful
0812 07:06:22.589553 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":155,"threads_started":1}
W20260812 07:06:22.592366 30062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:22.592468 30062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:22.592540 30041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:22.593380 30041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:22.599561 30041 catalog_manager.cc:1347] Generated new cluster ID: 8eeda51e310a42e2a89abd10f4d2771a
I20260812 07:06:22.599642 30041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:22.612913 30041 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:22.614212 30041 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:22.631258 30041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: Generated new TSK 0
I20260812 07:06:22.632076 30041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
Entry for principal kudu/127.27.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab.
I20260812 07:06:22.672039 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39299
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:22.789330 30074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:22.789573 30074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:22.789642 30074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:22.793656 30074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:22.793713 30074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:22.793802 30074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:06:22.798995 30074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.193.keytab
--principal=kudu/127.27.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:22.800069 30074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:22.801106 30074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:22.810774 30091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:22.810827 30074 server_base.cc:1034] running on GCE node
W20260812 07:06:22.810762 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:22.810762 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:22.811491 30074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:22.812063 30074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:22.813257 30074 hybrid_clock.cc:648] HybridClock initialized: now 1786518382813222 us; error 46 us; skew 500 ppm
Aug 12 07:06:22 dist-test-slave-w4v5 krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518382, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:22.816242 30074 init.cc:377] Logged in from keytab as kudu/127.27.110.193@KRBTEST.COM (short username kudu)
I20260812 07:06:22.817165 30074 webserver.cc:460] Webserver started at http://127.27.110.193:43133/ using document root <none> and password file <none>
I20260812 07:06:22.817755 30074 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:22.817854 30074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:22.818096 30074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:22.820324 30074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "e750b0f8be6d43fcb7fa262e627d73a0"
format_stamp: "Formatted at 2026-08-12 07:06:22 on dist-test-slave-w4v5"
server_key: "4755d03cb5cde45e276e062096f671fa"
server_key_iv: "a3a73564ad5bff1738600ec0cfc76153"
server_key_version: "encryptionkey@0"
I20260812 07:06:22.820948 30074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "e750b0f8be6d43fcb7fa262e627d73a0"
format_stamp: "Formatted at 2026-08-12 07:06:22 on dist-test-slave-w4v5"
server_key: "4755d03cb5cde45e276e062096f671fa"
server_key_iv: "a3a73564ad5bff1738600ec0cfc76153"
server_key_version: "encryptionkey@0"
I20260812 07:06:22.824836 30074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:06:22.827240 30106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:22.828459 30074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:22.828605 30074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "e750b0f8be6d43fcb7fa262e627d73a0"
format_stamp: "Formatted at 2026-08-12 07:06:22 on dist-test-slave-w4v5"
server_key: "4755d03cb5cde45e276e062096f671fa"
server_key_iv: "a3a73564ad5bff1738600ec0cfc76153"
server_key_version: "encryptionkey@0"
I20260812 07:06:22.828719 30074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:22.852084 30074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:22.854952 30074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:22.855186 30074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:22.855800 30074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:22.856637 30074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:22.856711 30074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:22.856794 30074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:22.856846 30074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:22.865918 30074 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:40379
I20260812 07:06:22.865952 30342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:40379 every 8 connection(s)
I20260812 07:06:22.866827 30074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:22.869587 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30074
I20260812 07:06:22.869724 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:22.870040 28091 external_mini_cluster.cc:1428] Setting key 6d7ffa169fe7ce740d442c0abcdc5bd0
Aug 12 07:06:22 dist-test-slave-w4v5 krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518382, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.193@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.194@KRBTEST.COM; defaulting to no policy
I20260812 07:06:22.881870 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:22.868696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:55509 (local address 127.27.110.254:39299)
0812 07:06:22.868842 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:22.868846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:22.869696 (+ 850us) server_negotiation.cc:407] Connection header received
0812 07:06:22.871227 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:22.871231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:22.871287 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:22.871399 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:22.872560 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:22.873090 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:22.874850 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:22.875167 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:22.875255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:22.878193 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:22.878201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:22.878203 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:22.878227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:22.879726 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:22.880247 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:22.880250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:22.880252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:22.880336 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:22.880692 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:22.880697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:22.880699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:22.880924 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:22.881037 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:22.881528 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 07:06:22.881654 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:06:22.882678 30347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:22.869078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39299 (local address 127.27.110.193:55509)
0812 07:06:22.869528 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:22.869571 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:22.870968 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:22.871579 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:22.871586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:22.871933 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:22.872407 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:22.872420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:22.873232 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:22.873238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:22.874707 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:22.874721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:22.875327 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:22.875334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:22.875577 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:22.876226 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:22.876251 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:22.878023 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:22.879851 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:22.879857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:22.879871 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:22.880138 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:22.880433 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:22.880437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:22.880439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:22.880588 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:22.881042 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:22.881051 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:22.881292 (+ 241us) client_negotiation.cc:769] Sending connection context
0812 07:06:22.881483 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 07:06:22.881823 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.194@KRBTEST.COM" created.
I20260812 07:06:22.884191 30343 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39299
I20260812 07:06:22.884495 30343 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:22.885131 30343 heartbeater.cc:511] Master 127.27.110.254:39299 requested a full tablet report, sending...
I20260812 07:06:22.887163 29939 ts_manager.cc:194] Registered new tserver with Master: e750b0f8be6d43fcb7fa262e627d73a0 (127.27.110.193:40379)
I20260812 07:06:22.888523 29939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.193@KRBTEST.COM'} at 127.27.110.193:55509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
Entry for principal kudu/127.27.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab.
I20260812 07:06:22.902485 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39299
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:23.016340 30352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:23.016607 30352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:23.016652 30352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:23.020800 30352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:23.020869 30352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:23.020962 30352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:06:23.026503 30352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.194.keytab
--principal=kudu/127.27.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:23.027673 30352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:23.028712 30352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:23.038817 30367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.038839 30364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.038790 30365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:23.039177 30352 server_base.cc:1034] running on GCE node
I20260812 07:06:23.039674 30352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:23.040287 30352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:23.041486 30352 hybrid_clock.cc:648] HybridClock initialized: now 1786518383041470 us; error 25 us; skew 500 ppm
Aug 12 07:06:23 dist-test-slave-w4v5 krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:23.044406 30352 init.cc:377] Logged in from keytab as kudu/127.27.110.194@KRBTEST.COM (short username kudu)
I20260812 07:06:23.045377 30352 webserver.cc:460] Webserver started at http://127.27.110.194:42415/ using document root <none> and password file <none>
I20260812 07:06:23.045944 30352 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:23.046005 30352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:23.046278 30352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:23.048509 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "44779ef7769b4aa49c8f55ab47e4baee"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "f1e9e1935cdc26a695e964d1a5a7a03d"
server_key_iv: "a8d2ac6a112b2d72d1af379969529a60"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.049108 30352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "44779ef7769b4aa49c8f55ab47e4baee"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "f1e9e1935cdc26a695e964d1a5a7a03d"
server_key_iv: "a8d2ac6a112b2d72d1af379969529a60"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.053275 30352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:06:23.055764 30379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.056977 30352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:23.057120 30352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "44779ef7769b4aa49c8f55ab47e4baee"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "f1e9e1935cdc26a695e964d1a5a7a03d"
server_key_iv: "a8d2ac6a112b2d72d1af379969529a60"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.057229 30352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:23.070058 30352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:23.073083 30352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:23.073285 30352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:23.073874 30352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:23.074702 30352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:23.074774 30352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.074846 30352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:23.074891 30352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.084317 30352 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:42285
I20260812 07:06:23.084350 30599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:42285 every 8 connection(s)
I20260812 07:06:23.085187 30352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:23.089891 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30352
I20260812 07:06:23.090041 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:23.090323 28091 external_mini_cluster.cc:1428] Setting key dbc3cbb976f60c8cbfc34efb8f8d8a17
Aug 12 07:06:23 dist-test-slave-w4v5 krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.194@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:23.100455 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.086916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:41009 (local address 127.27.110.254:39299)
0812 07:06:23.087116 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.087120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.088199 (+ 1079us) server_negotiation.cc:407] Connection header received
0812 07:06:23.089733 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.089737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.089795 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.089877 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:23.091404 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.092125 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.093329 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.093716 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.093840 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.096742 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:23.096746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:23.096748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:23.096770 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:23.098316 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:23.098854 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:23.098862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:23.098863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:23.098940 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:23.099330 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:23.099333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:23.099335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:23.099521 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:23.099640 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.100156 (+ 516us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.100281 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.27.110.195@KRBTEST.COM; defaulting to no policy
I20260812 07:06:23.101339 30604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.087373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39299 (local address 127.27.110.194:41009)
0812 07:06:23.088046 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.088092 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.089513 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.090088 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.090096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.090552 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:23.091239 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.091259 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.092278 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.092281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.093198 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.093208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.093898 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.093904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.094165 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.094797 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:23.094823 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:23.096571 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:23.098441 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:23.098447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:23.098462 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:23.098740 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:23.099041 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:23.099044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:23.099046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:23.099223 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:23.099638 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:23.099645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:23.099904 (+ 259us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.100097 (+ 193us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.100451 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":135,"threads_started":1}
I20260812 07:06:23.102552 30601 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39299
I20260812 07:06:23.102795 30601 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:23.103394 30601 heartbeater.cc:511] Master 127.27.110.254:39299 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.195@KRBTEST.COM" created.
I20260812 07:06:23.104591 29939 ts_manager.cc:194] Registered new tserver with Master: 44779ef7769b4aa49c8f55ab47e4baee (127.27.110.194:42285)
I20260812 07:06:23.105181 29939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.194@KRBTEST.COM'} at 127.27.110.194:41009
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
Entry for principal kudu/127.27.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab.
I20260812 07:06:23.122097 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39299
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:23.238001 30613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:23.238265 30613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:23.238312 30613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:23.242419 30613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:23.242482 30613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:23.242580 30613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:06:23.248135 30613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.195.keytab
--principal=kudu/127.27.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:23.249374 30613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:23.250402 30613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:23.260147 30630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.260164 30628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.260221 30625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:23.260490 30613 server_base.cc:1034] running on GCE node
I20260812 07:06:23.260932 30613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:23.261617 30613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:23.262782 30613 hybrid_clock.cc:648] HybridClock initialized: now 1786518383262767 us; error 25 us; skew 500 ppm
Aug 12 07:06:23 dist-test-slave-w4v5 krb5kdc[29839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:23.265595 30613 init.cc:377] Logged in from keytab as kudu/127.27.110.195@KRBTEST.COM (short username kudu)
I20260812 07:06:23.266656 30613 webserver.cc:460] Webserver started at http://127.27.110.195:35659/ using document root <none> and password file <none>
I20260812 07:06:23.267338 30613 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:23.267433 30613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:23.267728 30613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:23.269982 30613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "e4c2da14444f478e88aa001555661901"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "962bfdfa0f2ed9dd535074a87254db83"
server_key_iv: "3580cb07f4e235acead45f48d80e50fb"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.270627 30613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "e4c2da14444f478e88aa001555661901"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "962bfdfa0f2ed9dd535074a87254db83"
server_key_iv: "3580cb07f4e235acead45f48d80e50fb"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.274691 30613 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 07:06:23.277240 30640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.278478 30613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:23.278617 30613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "e4c2da14444f478e88aa001555661901"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "962bfdfa0f2ed9dd535074a87254db83"
server_key_iv: "3580cb07f4e235acead45f48d80e50fb"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.278759 30613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:23.290827 30613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:23.293799 30613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:23.294006 30613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:23.294584 30613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:23.295485 30613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:23.295567 30613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.295647 30613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:23.295692 30613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.304991 30613 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:45143
I20260812 07:06:23.305022 30839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:45143 every 8 connection(s)
I20260812 07:06:23.305887 30613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:23.309718 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30613
I20260812 07:06:23.309855 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:23.310153 28091 external_mini_cluster.cc:1428] Setting key bc01d7d02504f3f7797a5e82587ef1a9
Aug 12 07:06:23 dist-test-slave-w4v5 krb5kdc[29839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518383, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.195@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:23.319888 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.307749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:40199 (local address 127.27.110.254:39299)
0812 07:06:23.307900 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.307905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.308770 (+ 865us) server_negotiation.cc:407] Connection header received
0812 07:06:23.310260 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.310263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.310324 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.310441 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:23.311712 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.312329 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.313554 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.313781 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.313896 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.316441 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:23.316445 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:23.316452 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:23.316476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:23.317796 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:23.318318 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:23.318323 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:23.318324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:23.318406 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:23.318748 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:23.318752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:23.318754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:23.318938 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:23.319046 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.319604 (+ 558us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.319723 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:06:23.320710 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.308146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39299 (local address 127.27.110.195:40199)
0812 07:06:23.308620 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.308663 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.310030 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.310639 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.310646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.311001 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:23.311559 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.311573 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.312493 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.312499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.313418 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.313428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.313939 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.313943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.314118 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.314737 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:23.314755 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:23.316277 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:23.317908 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:23.317914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:23.317928 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:23.318195 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:23.318499 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:23.318502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:23.318504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:23.318648 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:23.319044 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:23.319051 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:23.319355 (+ 304us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.319549 (+ 194us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.319900 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260812 07:06:23.321884 30844 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39299
I20260812 07:06:23.322129 30844 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:23.322638 30844 heartbeater.cc:511] Master 127.27.110.254:39299 requested a full tablet report, sending...
I20260812 07:06:23.323823 29939 ts_manager.cc:194] Registered new tserver with Master: e4c2da14444f478e88aa001555661901 (127.27.110.195:45143)
I20260812 07:06:23.324447 29939 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.110.195@KRBTEST.COM'} at 127.27.110.195:40199
I20260812 07:06:23.335954 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:23.345089 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.338118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.27.110.254:39299)
0812 07:06:23.338281 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.338284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.338501 (+ 217us) server_negotiation.cc:407] Connection header received
0812 07:06:23.338595 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.338597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.338636 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.338717 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:23.339450 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.339952 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.340783 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.340983 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.341056 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.342373 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:23.342376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:23.342377 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:23.342399 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:23.343676 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:23.344054 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:23.344057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:23.344058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:23.344128 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:23.344372 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:23.344374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:23.344376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:23.344580 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:23.344682 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.344879 (+ 197us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.344968 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 07:06:23.347954 29939 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:23.351184 29939 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:23.365545 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.360084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.27.110.194:42285)
0812 07:06:23.360820 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.360837 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.360862 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:06:23.360943 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.360948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.361122 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.361296 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:23.361496 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.362598 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.364074 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.364774 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.364902 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.365035 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.365282 (+ 247us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.365389 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":162,"threads_started":1}
I20260812 07:06:23.365542 30869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.359585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42285 (local address 127.0.0.1:57566)
0812 07:06:23.360336 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.360348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.360433 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.361276 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.361279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.361300 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:23.361384 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.361390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.362743 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.362746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.363933 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.363941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.364915 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.364922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.365059 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.365101 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.365233 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.365367 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":3,"thread_start_us":94,"threads_started":1}
I20260812 07:06:23.365541 30864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.359372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40379 (local address 127.0.0.1:60378)
0812 07:06:23.359901 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.359950 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.360083 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.360977 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.360984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.361015 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:23.361162 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.361171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.362497 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.362500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.363904 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.363916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.364944 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.364947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.365277 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.365298 (+ 21us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.365361 (+ 63us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.365413 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":225,"thread_start_us":88,"threads_started":1}
I20260812 07:06:23.366048 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.360092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:45143 (local address 127.0.0.1:42778)
0812 07:06:23.361258 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.361273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.361360 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.361932 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.361935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.361954 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:23.362039 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.362044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.363327 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.363330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.364385 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.364395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.365659 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.365666 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.365752 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.365775 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.365845 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.365908 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"thread_start_us":99,"threads_started":1}
I20260812 07:06:23.366397 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.359534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.27.110.193:40379)
0812 07:06:23.360356 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.360373 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.360399 (+ 26us) server_negotiation.cc:407] Connection header received
0812 07:06:23.360532 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.360538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.360768 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.360977 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:23.361314 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.362363 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.364046 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.364797 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.365078 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.365209 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.365606 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.365884 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":132,"threads_started":1}
I20260812 07:06:23.366470 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.360682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.27.110.195:45143)
0812 07:06:23.361162 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.361173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.361460 (+ 287us) server_negotiation.cc:407] Connection header received
0812 07:06:23.361593 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.361598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.361774 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.361921 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:23.362170 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.363184 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.364535 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.365354 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.365471 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.365586 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.366240 (+ 654us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.366339 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":146,"threads_started":1}
I20260812 07:06:23.368780 30481 tablet_service.cc:1467] Processing CreateTablet for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a (DEFAULT_TABLE table=test-table [id=cb659c32dae7479899db18d566f65cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:23.369261 30205 tablet_service.cc:1467] Processing CreateTablet for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a (DEFAULT_TABLE table=test-table [id=cb659c32dae7479899db18d566f65cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:23.369262 30720 tablet_service.cc:1467] Processing CreateTablet for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a (DEFAULT_TABLE table=test-table [id=cb659c32dae7479899db18d566f65cd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:23.369880 30481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:23.370316 30205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:23.370316 30720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:23.376399 30876 tablet_bootstrap.cc:492] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Bootstrap starting.
I20260812 07:06:23.377801 30879 tablet_bootstrap.cc:492] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: Bootstrap starting.
I20260812 07:06:23.378051 30878 tablet_bootstrap.cc:492] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Bootstrap starting.
I20260812 07:06:23.379169 30876 tablet_bootstrap.cc:654] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:23.379865 30878 tablet_bootstrap.cc:654] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:23.380005 30879 tablet_bootstrap.cc:654] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:23.380699 30876 log.cc:824] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:23.381114 30878 log.cc:824] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:23.381295 30879 log.cc:824] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:23.383170 30876 tablet_bootstrap.cc:492] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: No bootstrap required, opened a new log
I20260812 07:06:23.383354 30879 tablet_bootstrap.cc:492] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: No bootstrap required, opened a new log
I20260812 07:06:23.383358 30876 ts_tablet_manager.cc:1397] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260812 07:06:23.383586 30879 ts_tablet_manager.cc:1397] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:06:23.383632 30878 tablet_bootstrap.cc:492] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: No bootstrap required, opened a new log
I20260812 07:06:23.383757 30878 ts_tablet_manager.cc:1397] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:06:23.386351 30878 raft_consensus.cc:348] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.386541 30878 raft_consensus.cc:374] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:23.386572 30878 raft_consensus.cc:729] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c2da14444f478e88aa001555661901, State: Initialized, Role: FOLLOWER
I20260812 07:06:23.386998 30878 consensus_queue.cc:260] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.387250 30879 raft_consensus.cc:348] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.387250 30876 raft_consensus.cc:348] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.387490 30879 raft_consensus.cc:374] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:23.387490 30876 raft_consensus.cc:374] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:23.387550 30876 raft_consensus.cc:729] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44779ef7769b4aa49c8f55ab47e4baee, State: Initialized, Role: FOLLOWER
I20260812 07:06:23.387588 30879 raft_consensus.cc:729] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e750b0f8be6d43fcb7fa262e627d73a0, State: Initialized, Role: FOLLOWER
I20260812 07:06:23.388037 30879 consensus_queue.cc:260] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.388151 30844 heartbeater.cc:503] Master 127.27.110.254:39299 was elected leader, sending a full tablet report...
I20260812 07:06:23.388296 30878 ts_tablet_manager.cc:1428] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:23.388463 30876 consensus_queue.cc:260] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.389019 30879 ts_tablet_manager.cc:1428] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:23.389119 30601 heartbeater.cc:503] Master 127.27.110.254:39299 was elected leader, sending a full tablet report...
I20260812 07:06:23.389317 30343 heartbeater.cc:503] Master 127.27.110.254:39299 was elected leader, sending a full tablet report...
I20260812 07:06:23.390702 30876 ts_tablet_manager.cc:1428] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260812 07:06:23.417275 30889 raft_consensus.cc:482] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:23.417491 30889 raft_consensus.cc:504] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.418637 30889 leader_election.cc:288] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44779ef7769b4aa49c8f55ab47e4baee (127.27.110.194:42285), e4c2da14444f478e88aa001555661901 (127.27.110.195:45143)
I20260812 07:06:23.422886 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.419005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:43091 (local address 127.27.110.195:45143)
0812 07:06:23.419172 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.419178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.419697 (+ 519us) server_negotiation.cc:407] Connection header received
0812 07:06:23.419850 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.419855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.419909 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.419996 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:23.420308 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.420818 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.421861 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.422257 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.422393 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.422446 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.422669 (+ 223us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.422752 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 07:06:23.422943 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.419294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:42093 (local address 127.27.110.194:42285)
0812 07:06:23.419444 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.419450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.419469 (+ 19us) server_negotiation.cc:407] Connection header received
0812 07:06:23.419608 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.419615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.419688 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.419770 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:23.420111 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.420709 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.421820 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.422243 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.422365 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.422421 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.422749 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.422835 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 07:06:23.422938 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.418939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:45143 (local address 127.27.110.193:43091)
0812 07:06:23.419590 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.419604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.419705 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.420016 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.420018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.420033 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:23.420195 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.420202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.420929 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.420932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.421713 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.421721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.422397 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.422400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.422526 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.422562 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.422623 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.422700 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":3,"thread_start_us":143,"threads_started":1}
I20260812 07:06:23.422948 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.418905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:42285 (local address 127.27.110.193:42093)
0812 07:06:23.419365 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.419383 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:23.419503 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:23.419806 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:23.419811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:23.419834 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:23.419989 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.419996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.420822 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.420825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.421681 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:23.421691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.422383 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.422387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:23.422564 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.422584 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 07:06:23.422720 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 07:06:23.422785 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":64,"thread_start_us":175,"threads_started":1}
I20260812 07:06:23.423797 30519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d53e2c34f4b3c9083d6cf2a23cf0a" candidate_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44779ef7769b4aa49c8f55ab47e4baee" is_pre_election: true
I20260812 07:06:23.424118 30519 raft_consensus.cc:2454] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e750b0f8be6d43fcb7fa262e627d73a0 in term 0.
I20260812 07:06:23.424206 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d53e2c34f4b3c9083d6cf2a23cf0a" candidate_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c2da14444f478e88aa001555661901" is_pre_election: true
I20260812 07:06:23.424528 30765 raft_consensus.cc:2454] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e750b0f8be6d43fcb7fa262e627d73a0 in term 0.
I20260812 07:06:23.424633 30111 leader_election.cc:302] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44779ef7769b4aa49c8f55ab47e4baee, e750b0f8be6d43fcb7fa262e627d73a0; no voters:
I20260812 07:06:23.424966 30889 raft_consensus.cc:2781] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:23.425065 30889 raft_consensus.cc:482] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:23.425132 30889 raft_consensus.cc:3037] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:23.426290 30889 raft_consensus.cc:504] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.426640 30889 leader_election.cc:288] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [CANDIDATE]: Term 1 election: Requested vote from peers 44779ef7769b4aa49c8f55ab47e4baee (127.27.110.194:42285), e4c2da14444f478e88aa001555661901 (127.27.110.195:45143)
I20260812 07:06:23.427001 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d53e2c34f4b3c9083d6cf2a23cf0a" candidate_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c2da14444f478e88aa001555661901"
I20260812 07:06:23.427001 30519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f26d53e2c34f4b3c9083d6cf2a23cf0a" candidate_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44779ef7769b4aa49c8f55ab47e4baee"
I20260812 07:06:23.427151 30765 raft_consensus.cc:3037] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:23.427170 30519 raft_consensus.cc:3037] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:23.428500 30765 raft_consensus.cc:2454] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e750b0f8be6d43fcb7fa262e627d73a0 in term 1.
I20260812 07:06:23.428500 30519 raft_consensus.cc:2454] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e750b0f8be6d43fcb7fa262e627d73a0 in term 1.
I20260812 07:06:23.428825 30107 leader_election.cc:302] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44779ef7769b4aa49c8f55ab47e4baee, e4c2da14444f478e88aa001555661901, e750b0f8be6d43fcb7fa262e627d73a0; no voters:
I20260812 07:06:23.429037 30889 raft_consensus.cc:2781] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:23.429239 30889 raft_consensus.cc:686] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 1 LEADER]: Becoming Leader. State: Replica: e750b0f8be6d43fcb7fa262e627d73a0, State: Running, Role: LEADER
I20260812 07:06:23.429585 30889 consensus_queue.cc:237] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } }
I20260812 07:06:23.432936 29939 catalog_manager.cc:5495] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e750b0f8be6d43fcb7fa262e627d73a0 (127.27.110.193). New cstate: current_term: 1 leader_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e750b0f8be6d43fcb7fa262e627d73a0" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 40379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:23.462821 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.459123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60382 (local address 127.27.110.193:40379)
0812 07:06:23.459260 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.459265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.459281 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:06:23.459405 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.459408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.459477 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.459595 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:23.459862 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.460384 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.461315 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.461557 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.461687 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.462010 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:23.462421 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:23.462563 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.462604 (+ 41us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.462675 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260812 07:06:23.468734 30519 raft_consensus.cc:1261] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 1 FOLLOWER]: Refusing update from remote peer e750b0f8be6d43fcb7fa262e627d73a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:23.468735 30765 raft_consensus.cc:1261] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 1 FOLLOWER]: Refusing update from remote peer e750b0f8be6d43fcb7fa262e627d73a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:23.469465 30893 consensus_queue.cc:1035] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c2da14444f478e88aa001555661901" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 45143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:23.469645 30889 consensus_queue.cc:1035] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44779ef7769b4aa49c8f55ab47e4baee" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 42285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:23.476532 30899 mvcc.cc:205] Tried to move back new op lower bound from 7317579298680266752 to 7317579298530197504. Current Snapshot: MvccSnapshot[applied={T|T < 7317579298680266752}]
I20260812 07:06:23.477753 30900 mvcc.cc:205] Tried to move back new op lower bound from 7317579298680266752 to 7317579298530197504. Current Snapshot: MvccSnapshot[applied={T|T < 7317579298680266752}]
I20260812 07:06:23.485580 29936 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:23.485749 29936 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:23.488252 29936 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1360bddbac634f32b00cb0f64b371614: Sending DeleteTablet for 3 replicas of tablet f26d53e2c34f4b3c9083d6cf2a23cf0a
I20260812 07:06:23.489100 30720 tablet_service.cc:1514] Processing DeleteTablet for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:23 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:42778
I20260812 07:06:23.489097 30481 tablet_service.cc:1514] Processing DeleteTablet for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:23 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:57566
I20260812 07:06:23.489609 30915 tablet_replica.cc:330] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: stopping tablet replica
I20260812 07:06:23.489617 30205 tablet_service.cc:1514] Processing DeleteTablet for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:23 UTC) from {username='kudu', principal='kudu/127.27.110.254@KRBTEST.COM'} at 127.0.0.1:60378
I20260812 07:06:23.490029 30915 raft_consensus.cc:2229] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:23.490204 30917 tablet_replica.cc:330] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0: stopping tablet replica
I20260812 07:06:23.490355 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30074
I20260812 07:06:23.490525 30915 raft_consensus.cc:2258] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:23.490525 30917 raft_consensus.cc:2229] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e750b0f8be6d43fcb7fa262e627d73a0 [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:23.491142 30916 tablet_replica.cc:330] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: stopping tablet replica
I20260812 07:06:23.492134 30915 ts_tablet_manager.cc:1905] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:23.495357 30915 ts_tablet_manager.cc:1918] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:23.495452 30915 log.cc:1196] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/f26d53e2c34f4b3c9083d6cf2a23cf0a
I20260812 07:06:23.495749 30915 ts_tablet_manager.cc:1939] T f26d53e2c34f4b3c9083d6cf2a23cf0a P 44779ef7769b4aa49c8f55ab47e4baee: Deleting consensus metadata
I20260812 07:06:23.496137 30916 raft_consensus.cc:2229] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:23.496402 30916 raft_consensus.cc:2258] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:23.498802 30916 ts_tablet_manager.cc:1905] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:23.498867 29913 catalog_manager.cc:4841] TS 44779ef7769b4aa49c8f55ab47e4baee (127.27.110.194:42285): tablet f26d53e2c34f4b3c9083d6cf2a23cf0a (table test-table [id=cb659c32dae7479899db18d566f65cd1]) successfully deleted
I20260812 07:06:23.500928 30916 ts_tablet_manager.cc:1918] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:23.501055 30916 log.cc:1196] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/f26d53e2c34f4b3c9083d6cf2a23cf0a
I20260812 07:06:23.501358 30916 ts_tablet_manager.cc:1939] T f26d53e2c34f4b3c9083d6cf2a23cf0a P e4c2da14444f478e88aa001555661901: Deleting consensus metadata
I20260812 07:06:23.502460 29908 catalog_manager.cc:4841] TS e4c2da14444f478e88aa001555661901 (127.27.110.195:45143): tablet f26d53e2c34f4b3c9083d6cf2a23cf0a (table test-table [id=cb659c32dae7479899db18d566f65cd1]) successfully deleted
W20260812 07:06:23.504235 29908 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:40379 (error 108)
I20260812 07:06:23.504436 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30352
I20260812 07:06:23.508085 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.507745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:40379 (local address 127.0.0.1:60392)
0812 07:06:23.507923 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:23.507989 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260812 07:06:23.508248 29908 catalog_manager.cc:4584] TS e750b0f8be6d43fcb7fa262e627d73a0 (127.27.110.193:40379): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f26d53e2c34f4b3c9083d6cf2a23cf0a: Network error: Client connection negotiation failed: client connection to 127.27.110.193:40379: connect: Connection refused (error 111)
I20260812 07:06:23.525764 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30613
I20260812 07:06:23.542719 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 29873
2026-08-12T07:06:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2681 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T07:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:23Z Disabled control of system clock
I20260812 07:06:23.579777 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39627
--webserver_interface=127.27.110.254
--webserver_port=0
--builtin_ntp_servers=127.27.110.212:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:06:23.692974 30926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:23.693253 30926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:23.693292 30926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:23.697404 30926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:23.697463 30926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:23.697477 30926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:23.697494 30926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:23.697510 30926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:23.702342 30926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:23.703511 30926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:23.705013 30926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:23.713946 30943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.713946 30948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.713999 30944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:23.714238 30926 server_base.cc:1034] running on GCE node
I20260812 07:06:23.715304 30926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:23.716432 30926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:23.717643 30926 hybrid_clock.cc:648] HybridClock initialized: now 1786518383717609 us; error 47 us; skew 500 ppm
I20260812 07:06:23.719708 30926 webserver.cc:460] Webserver started at http://127.27.110.254:39407/ using document root <none> and password file <none>
I20260812 07:06:23.720307 30926 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:23.720435 30926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:23.720672 30926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:23.722795 30926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "3d8c6336d9e34824b04398c591f12cc8"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "f95c4704cab7faaeb544a47a3d068148"
server_key_iv: "b0ae9918c080e49a1e4d7ff424cec187"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.723394 30926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "3d8c6336d9e34824b04398c591f12cc8"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "f95c4704cab7faaeb544a47a3d068148"
server_key_iv: "b0ae9918c080e49a1e4d7ff424cec187"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.727337 30926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:23.729846 30956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.731333 30926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:23.731479 30926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "3d8c6336d9e34824b04398c591f12cc8"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "f95c4704cab7faaeb544a47a3d068148"
server_key_iv: "b0ae9918c080e49a1e4d7ff424cec187"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.731602 30926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:23.748898 30926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:23.749593 30926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:23.749807 30926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:23.757491 30926 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39627
I20260812 07:06:23.757542 31053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39627 every 8 connection(s)
I20260812 07:06:23.758486 30926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:23.761766 31056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:23.766312 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 30926
I20260812 07:06:23.766449 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:23.766790 28091 external_mini_cluster.cc:1428] Setting key d3766d2ee09dd0849f6e8e50172cab62
I20260812 07:06:23.767786 31056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: Bootstrap starting.
I20260812 07:06:23.770290 31056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:23.771055 31056 log.cc:824] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:23.773072 31056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: No bootstrap required, opened a new log
I20260812 07:06:23.776465 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:23.768295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.27.110.254:39627)
0812 07:06:23.768972 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:23.768988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:23.769023 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:06:23.769629 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:23.769644 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:23.770028 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:23.770396 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:23.770855 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.772114 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.773045 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:23.773573 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:23.773743 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:23.774648 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:23.774656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:23.774676 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:23.775044 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:23.775166 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:23.775190 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:23.775295 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:23.775412 (+ 117us) server_negotiation.cc:299] Negotiation successful
0812 07:06:23.775672 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":136,"threads_started":1}
I20260812 07:06:23.776953 31056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8c6336d9e34824b04398c591f12cc8" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39627 } }
I20260812 07:06:23.777226 31056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:23.777282 31056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8c6336d9e34824b04398c591f12cc8, State: Initialized, Role: FOLLOWER
I20260812 07:06:23.777711 31056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8c6336d9e34824b04398c591f12cc8" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39627 } }
I20260812 07:06:23.777834 31056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:23.777921 31056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:23.778043 31056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:23.779134 31056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8c6336d9e34824b04398c591f12cc8" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39627 } }
I20260812 07:06:23.779486 31056 leader_election.cc:302] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d8c6336d9e34824b04398c591f12cc8; no voters:
I20260812 07:06:23.779790 31056 leader_election.cc:288] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:23.779912 31068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:23.780161 31068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [term 1 LEADER]: Becoming Leader. State: Replica: 3d8c6336d9e34824b04398c591f12cc8, State: Running, Role: LEADER
I20260812 07:06:23.780467 31068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8c6336d9e34824b04398c591f12cc8" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39627 } }
I20260812 07:06:23.781041 31056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:23.782186 31070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d8c6336d9e34824b04398c591f12cc8. Latest consensus state: current_term: 1 leader_uuid: "3d8c6336d9e34824b04398c591f12cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8c6336d9e34824b04398c591f12cc8" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39627 } } }
I20260812 07:06:23.782497 31070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:23.782685 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d8c6336d9e34824b04398c591f12cc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d8c6336d9e34824b04398c591f12cc8" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39627 } } }
I20260812 07:06:23.782789 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8 [sys.catalog]: This master's current role is: LEADER
W20260812 07:06:23.786181 31100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:23.786262 31100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:23.786340 31090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:23.787020 31090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:23.792055 31090 catalog_manager.cc:1347] Generated new cluster ID: 9143b75d691d450eba759723e48db2c1
I20260812 07:06:23.792119 31090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:23.805583 31090 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:23.806821 31090 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:23.822005 31090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: Generated new TSK 0
I20260812 07:06:23.822860 31090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:23.830178 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.193:0
--local_ip_for_outbound_sockets=127.27.110.193
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39627
--builtin_ntp_servers=127.27.110.212:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 07:06:23.959643 31104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:23.959895 31104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:23.959931 31104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:23.963925 31104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:23.963975 31104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:23.964026 31104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.193
I20260812 07:06:23.968689 31104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.110.193
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.193
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:23.969746 31104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:23.971257 31104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:23.981009 31120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:23.981093 31104 server_base.cc:1034] running on GCE node
W20260812 07:06:23.981010 31119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:23.980999 31122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:23.981678 31104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:23.982348 31104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:23.983579 31104 hybrid_clock.cc:648] HybridClock initialized: now 1786518383983550 us; error 40 us; skew 500 ppm
I20260812 07:06:23.985512 31104 webserver.cc:460] Webserver started at http://127.27.110.193:34487/ using document root <none> and password file <none>
I20260812 07:06:23.986090 31104 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:23.986186 31104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:23.986423 31104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:23.988543 31104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data/instance:
uuid: "568404922b88405380d6925d581389c8"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "682c4cc0c7ecef7d4d6c09110f3c21ad"
server_key_iv: "86a5960790e8ae84b2a95be837946935"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.989121 31104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance:
uuid: "568404922b88405380d6925d581389c8"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "682c4cc0c7ecef7d4d6c09110f3c21ad"
server_key_iv: "86a5960790e8ae84b2a95be837946935"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.992921 31104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:06:23.995438 31133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:23.996819 31104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:06:23.996968 31104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal
uuid: "568404922b88405380d6925d581389c8"
format_stamp: "Formatted at 2026-08-12 07:06:23 on dist-test-slave-w4v5"
server_key: "682c4cc0c7ecef7d4d6c09110f3c21ad"
server_key_iv: "86a5960790e8ae84b2a95be837946935"
server_key_version: "encryptionkey@0"
I20260812 07:06:23.997080 31104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:24.016671 31104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:24.017397 31104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:24.017613 31104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:24.018240 31104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:24.019294 31104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:24.019366 31104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.019438 31104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:24.019491 31104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.028882 31104 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.193:43661
I20260812 07:06:24.028934 31338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.193:43661 every 8 connection(s)
I20260812 07:06:24.029834 31104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/data/info.pb
I20260812 07:06:24.038822 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 31104
I20260812 07:06:24.038977 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-0/wal/instance
I20260812 07:06:24.039340 28091 external_mini_cluster.cc:1428] Setting key 420666eaedc6c5576746233b25160b87
I20260812 07:06:24.039378 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.031798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.193:57293 (local address 127.27.110.254:39627)
0812 07:06:24.031940 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.031944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.033047 (+ 1103us) server_negotiation.cc:407] Connection header received
0812 07:06:24.034486 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.034489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.034550 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.034661 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:24.035627 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.036187 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.037330 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.037561 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.037676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.038833 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:24.038836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:24.038838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:24.038851 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:24.038871 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:24.038879 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:24.038988 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.039208 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.039265 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:06:24.040406 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.032217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39627 (local address 127.27.110.193:57293)
0812 07:06:24.032898 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.032939 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.034262 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.034837 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.034844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.034926 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:24.035476 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.035490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.036331 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.036334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.037202 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.037211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.037727 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.037731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.037898 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.038474 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:24.038475 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:24.038536 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:24.038659 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:24.039009 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:24.039021 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:24.039030 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.039200 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.039498 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260812 07:06:24.041563 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.194:0
--local_ip_for_outbound_sockets=127.27.110.194
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39627
--builtin_ntp_servers=127.27.110.212:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:06:24.041671 31339 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39627
I20260812 07:06:24.041862 31339 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:24.042300 31339 heartbeater.cc:511] Master 127.27.110.254:39627 requested a full tablet report, sending...
I20260812 07:06:24.043725 30985 ts_manager.cc:194] Registered new tserver with Master: 568404922b88405380d6925d581389c8 (127.27.110.193:43661)
I20260812 07:06:24.044867 30985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.193:57293
W20260812 07:06:24.158824 31349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:24.159132 31349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:24.159197 31349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:24.163040 31349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:24.163101 31349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:24.163156 31349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.194
I20260812 07:06:24.167753 31349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.110.194
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.194
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:24.168732 31349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:24.170039 31349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:24.179935 31361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:24.179920 31365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:24.179925 31358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:24.180305 31349 server_base.cc:1034] running on GCE node
I20260812 07:06:24.180733 31349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:24.181456 31349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:24.182643 31349 hybrid_clock.cc:648] HybridClock initialized: now 1786518384182633 us; error 31 us; skew 500 ppm
I20260812 07:06:24.184577 31349 webserver.cc:460] Webserver started at http://127.27.110.194:41617/ using document root <none> and password file <none>
I20260812 07:06:24.185200 31349 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:24.185299 31349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:24.185539 31349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:24.187656 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data/instance:
uuid: "8cc2252caeb046d6b5339c4fdb55806e"
format_stamp: "Formatted at 2026-08-12 07:06:24 on dist-test-slave-w4v5"
server_key: "08b291f98c9ecc9bafa97ac5e7b551b9"
server_key_iv: "82fdfd54e1bc5dfc3c99b79c23681c83"
server_key_version: "encryptionkey@0"
I20260812 07:06:24.188258 31349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance:
uuid: "8cc2252caeb046d6b5339c4fdb55806e"
format_stamp: "Formatted at 2026-08-12 07:06:24 on dist-test-slave-w4v5"
server_key: "08b291f98c9ecc9bafa97ac5e7b551b9"
server_key_iv: "82fdfd54e1bc5dfc3c99b79c23681c83"
server_key_version: "encryptionkey@0"
I20260812 07:06:24.191956 31349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 07:06:24.194420 31372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.195652 31349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260812 07:06:24.195784 31349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal
uuid: "8cc2252caeb046d6b5339c4fdb55806e"
format_stamp: "Formatted at 2026-08-12 07:06:24 on dist-test-slave-w4v5"
server_key: "08b291f98c9ecc9bafa97ac5e7b551b9"
server_key_iv: "82fdfd54e1bc5dfc3c99b79c23681c83"
server_key_version: "encryptionkey@0"
I20260812 07:06:24.195894 31349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:24.208683 31349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:24.209375 31349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:24.209580 31349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:24.210201 31349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:24.211259 31349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:24.211329 31349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.211400 31349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:24.211447 31349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.220274 31349 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.194:45997
I20260812 07:06:24.220353 31614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.194:45997 every 8 connection(s)
I20260812 07:06:24.221230 31349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/data/info.pb
I20260812 07:06:24.229492 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 31349
I20260812 07:06:24.229696 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal/instance
I20260812 07:06:24.229993 28091 external_mini_cluster.cc:1428] Setting key 2298bbd3a6b4e6b1858350efcd9f7b93
I20260812 07:06:24.230970 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.223113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:40821 (local address 127.27.110.254:39627)
0812 07:06:24.223241 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.223244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.224388 (+ 1144us) server_negotiation.cc:407] Connection header received
0812 07:06:24.225873 (+ 1485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.225877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.225933 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.226045 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:24.226952 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.227524 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.228778 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.229004 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.229124 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.230407 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:24.230411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:24.230413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:24.230427 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:24.230465 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:24.230491 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:24.230579 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.230758 (+ 179us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.230825 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 07:06:24.231884 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.223533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39627 (local address 127.27.110.194:40821)
0812 07:06:24.224226 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.224267 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.225654 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.226235 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.226243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.226327 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:24.226809 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.226821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.227675 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.227678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.228650 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.228660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.229168 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.229172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.229399 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.230114 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:24.230121 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:24.230184 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:24.230251 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:24.230596 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:24.230606 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:24.230616 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.230756 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.231051 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260812 07:06:24.232338 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.110.195:0
--local_ip_for_outbound_sockets=127.27.110.195
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39627
--builtin_ntp_servers=127.27.110.212:33275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 07:06:24.233026 31618 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39627
I20260812 07:06:24.233306 31618 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:24.233963 31618 heartbeater.cc:511] Master 127.27.110.254:39627 requested a full tablet report, sending...
I20260812 07:06:24.235276 30985 ts_manager.cc:194] Registered new tserver with Master: 8cc2252caeb046d6b5339c4fdb55806e (127.27.110.194:45997)
I20260812 07:06:24.236060 30985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.194:40821
W20260812 07:06:24.349977 31625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:24.350221 31625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:24.350258 31625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:24.354338 31625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:24.354393 31625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:24.354444 31625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.110.195
I20260812 07:06:24.358964 31625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:33275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.110.195
--webserver_port=0
--tserver_master_addrs=127.27.110.254:39627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.110.195
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:24.359966 31625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:24.361270 31625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:24.371011 31639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:24.371047 31640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:24.371206 31625 server_base.cc:1034] running on GCE node
W20260812 07:06:24.371418 31647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:24.371805 31625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:24.372476 31625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:24.373665 31625 hybrid_clock.cc:648] HybridClock initialized: now 1786518384373651 us; error 26 us; skew 500 ppm
I20260812 07:06:24.375569 31625 webserver.cc:460] Webserver started at http://127.27.110.195:33925/ using document root <none> and password file <none>
I20260812 07:06:24.376152 31625 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:24.376250 31625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:24.376488 31625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:24.378644 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data/instance:
uuid: "18ceb566bbd14163a2965689beedfc3c"
format_stamp: "Formatted at 2026-08-12 07:06:24 on dist-test-slave-w4v5"
server_key: "41c79495178134f6abe427384103ddc4"
server_key_iv: "dfc0bd3711123324a029237c61a07640"
server_key_version: "encryptionkey@0"
I20260812 07:06:24.379280 31625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance:
uuid: "18ceb566bbd14163a2965689beedfc3c"
format_stamp: "Formatted at 2026-08-12 07:06:24 on dist-test-slave-w4v5"
server_key: "41c79495178134f6abe427384103ddc4"
server_key_iv: "dfc0bd3711123324a029237c61a07640"
server_key_version: "encryptionkey@0"
I20260812 07:06:24.383246 31625 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:24.385797 31656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.387157 31625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:24.387288 31625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal
uuid: "18ceb566bbd14163a2965689beedfc3c"
format_stamp: "Formatted at 2026-08-12 07:06:24 on dist-test-slave-w4v5"
server_key: "41c79495178134f6abe427384103ddc4"
server_key_iv: "dfc0bd3711123324a029237c61a07640"
server_key_version: "encryptionkey@0"
I20260812 07:06:24.387400 31625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:24.401364 31625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:24.402089 31625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:24.402298 31625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:24.402870 31625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 07:06:24.403823 31625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 07:06:24.403919 31625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.403992 31625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 07:06:24.404029 31625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:24.413230 31625 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.195:44927
I20260812 07:06:24.413262 31905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.195:44927 every 8 connection(s)
I20260812 07:06:24.414111 31625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/data/info.pb
I20260812 07:06:24.420807 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 31625
I20260812 07:06:24.420940 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal/instance
I20260812 07:06:24.421237 28091 external_mini_cluster.cc:1428] Setting key 6bedbebf3dab1edc81ce0d126b29f7ee
I20260812 07:06:24.424103 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.415985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.195:59409 (local address 127.27.110.254:39627)
0812 07:06:24.416132 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.416136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.417212 (+ 1076us) server_negotiation.cc:407] Connection header received
0812 07:06:24.418635 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.418638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.418697 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.418806 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:24.419712 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.420257 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.421840 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.422179 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.422291 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.423652 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:24.423655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:24.423664 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:24.423676 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:24.423701 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:24.423717 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:24.423783 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.423946 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.423996 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 07:06:24.425004 31911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.416376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:39627 (local address 127.27.110.195:59409)
0812 07:06:24.417053 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.417093 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.418409 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.418980 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.418987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.419099 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:24.419572 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.419584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.420395 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.420401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.421689 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.421699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.422352 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.422356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.422587 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.423340 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 07:06:24.423343 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:24.423396 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 07:06:24.423501 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:24.423809 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:24.423817 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:24.423821 (+ 4us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.423923 (+ 102us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.424148 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":142,"threads_started":1}
I20260812 07:06:24.426198 31907 heartbeater.cc:347] Connected to a master server at 127.27.110.254:39627
I20260812 07:06:24.426429 31907 heartbeater.cc:464] Registering TS with master...
I20260812 07:06:24.426944 31907 heartbeater.cc:511] Master 127.27.110.254:39627 requested a full tablet report, sending...
I20260812 07:06:24.428035 30985 ts_manager.cc:194] Registered new tserver with Master: 18ceb566bbd14163a2965689beedfc3c (127.27.110.195:44927)
I20260812 07:06:24.428654 30985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.110.195:59409
I20260812 07:06:24.435256 28091 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 07:06:24.440864 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.437303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.27.110.254:39627)
0812 07:06:24.437424 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.437427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.437611 (+ 184us) server_negotiation.cc:407] Connection header received
0812 07:06:24.437722 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.437724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.437761 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.437829 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:24.438075 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.438552 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.439429 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.439642 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.439727 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.440450 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:24.440454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:24.440457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:24.440470 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:24.440498 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:24.440506 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:24.440574 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.440701 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.440752 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:06:24.443709 30985 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 07:06:24.446682 30985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 07:06:24.460793 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.455735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.194:45997 (local address 127.0.0.1:57854)
0812 07:06:24.455975 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.455999 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.456095 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.456990 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.456996 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.457016 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:24.457100 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.457105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.458117 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.458122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.459363 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.459374 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.460281 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.460285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.460385 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.460419 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.460517 (+ 98us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.460631 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":113,"threads_started":1}
I20260812 07:06:24.460800 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.456186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.27.110.194:45997)
0812 07:06:24.456499 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.456512 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.456537 (+ 25us) server_negotiation.cc:407] Connection header received
0812 07:06:24.456612 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.456617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.456801 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.457010 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:24.457195 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.458002 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.459516 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.460161 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.460293 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.460476 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.460541 (+ 65us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.460634 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":131,"threads_started":1}
I20260812 07:06:24.461397 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.455149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.27.110.193:43661)
0812 07:06:24.455622 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.455639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.455884 (+ 245us) server_negotiation.cc:407] Connection header received
0812 07:06:24.456108 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.456114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.456356 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.456489 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:24.457032 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.458251 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.459701 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.460486 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.460611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.460793 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.461134 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.461232 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":117,"threads_started":1}
I20260812 07:06:24.461447 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.455402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44927 (local address 127.0.0.1:36620)
0812 07:06:24.455778 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.455794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.455921 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.457158 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.457162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.457180 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:24.457276 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.457281 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.458574 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.458577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.459727 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.459736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.461076 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.461080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.461168 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.461190 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.461266 (+ 76us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.461328 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":95,"threads_started":1}
I20260812 07:06:24.461813 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.454985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:43661 (local address 127.0.0.1:48870)
0812 07:06:24.455375 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.455793 (+ 418us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.455943 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.456708 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.456717 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.456746 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:24.456884 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.456893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.458386 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.458389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.459521 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.459533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.460862 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.460866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.460987 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.461007 (+ 20us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.461610 (+ 603us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.461694 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":75,"thread_start_us":108,"threads_started":1}
I20260812 07:06:24.461815 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.456172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.27.110.195:44927)
0812 07:06:24.456636 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.456647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.456667 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:06:24.456733 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.456737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.456894 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.457023 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:24.457403 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.458459 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.459852 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.460615 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.460723 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.461213 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.461290 (+ 77us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.461380 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":156,"threads_started":1}
I20260812 07:06:24.463932 31478 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e9f10fc6a45a9a7c3baf23e934721 (DEFAULT_TABLE table=test-table [id=1a76dff520d649a881094fd0c66e7bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:24.464879 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e9f10fc6a45a9a7c3baf23e934721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:24.464540 31218 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e9f10fc6a45a9a7c3baf23e934721 (DEFAULT_TABLE table=test-table [id=1a76dff520d649a881094fd0c66e7bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:24.464547 31754 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e9f10fc6a45a9a7c3baf23e934721 (DEFAULT_TABLE table=test-table [id=1a76dff520d649a881094fd0c66e7bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 07:06:24.465545 31754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e9f10fc6a45a9a7c3baf23e934721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:24.465545 31218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e9f10fc6a45a9a7c3baf23e934721. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:24.471525 31939 tablet_bootstrap.cc:492] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Bootstrap starting.
I20260812 07:06:24.472582 31943 tablet_bootstrap.cc:492] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Bootstrap starting.
I20260812 07:06:24.474056 31939 tablet_bootstrap.cc:654] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:24.474669 31943 tablet_bootstrap.cc:654] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:24.474960 31939 log.cc:824] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:24.475549 31943 log.cc:824] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:24.475697 31941 tablet_bootstrap.cc:492] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: Bootstrap starting.
I20260812 07:06:24.477108 31939 tablet_bootstrap.cc:492] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: No bootstrap required, opened a new log
I20260812 07:06:24.477317 31939 ts_tablet_manager.cc:1397] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260812 07:06:24.477717 31943 tablet_bootstrap.cc:492] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: No bootstrap required, opened a new log
I20260812 07:06:24.477792 31941 tablet_bootstrap.cc:654] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:24.477902 31943 ts_tablet_manager.cc:1397] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260812 07:06:24.478632 31941 log.cc:824] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:24.480573 31941 tablet_bootstrap.cc:492] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: No bootstrap required, opened a new log
I20260812 07:06:24.480796 31941 ts_tablet_manager.cc:1397] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260812 07:06:24.481017 31939 raft_consensus.cc:348] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.481261 31939 raft_consensus.cc:374] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:24.481313 31939 raft_consensus.cc:729] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cc2252caeb046d6b5339c4fdb55806e, State: Initialized, Role: FOLLOWER
I20260812 07:06:24.481855 31939 consensus_queue.cc:260] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.482192 31943 raft_consensus.cc:348] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.482427 31943 raft_consensus.cc:374] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:24.482472 31943 raft_consensus.cc:729] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ceb566bbd14163a2965689beedfc3c, State: Initialized, Role: FOLLOWER
I20260812 07:06:24.482489 31618 heartbeater.cc:503] Master 127.27.110.254:39627 was elected leader, sending a full tablet report...
I20260812 07:06:24.482910 31939 ts_tablet_manager.cc:1428] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260812 07:06:24.483331 31943 consensus_queue.cc:260] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.484050 31907 heartbeater.cc:503] Master 127.27.110.254:39627 was elected leader, sending a full tablet report...
I20260812 07:06:24.484321 31943 ts_tablet_manager.cc:1428] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260812 07:06:24.485773 31941 raft_consensus.cc:348] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.486075 31941 raft_consensus.cc:374] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:24.486140 31941 raft_consensus.cc:729] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 568404922b88405380d6925d581389c8, State: Initialized, Role: FOLLOWER
I20260812 07:06:24.486554 31941 consensus_queue.cc:260] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.487174 31339 heartbeater.cc:503] Master 127.27.110.254:39627 was elected leader, sending a full tablet report...
I20260812 07:06:24.488338 31941 ts_tablet_manager.cc:1428] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20260812 07:06:24.531066 31342 tablet.cc:2369] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 07:06:24.615543 31953 raft_consensus.cc:482] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 07:06:24.615846 31953 raft_consensus.cc:504] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.617106 31953 leader_election.cc:288] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 568404922b88405380d6925d581389c8 (127.27.110.193:43661), 18ceb566bbd14163a2965689beedfc3c (127.27.110.195:44927)
I20260812 07:06:24.622041 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.617460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:38665 (local address 127.27.110.193:43661)
0812 07:06:24.617622 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.617630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.617653 (+ 23us) server_negotiation.cc:407] Connection header received
0812 07:06:24.617717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.617720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.617779 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.617861 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:24.618208 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.619066 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.620735 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.621348 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.621489 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.621537 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.621827 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.621904 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 07:06:24.622059 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.617629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.110.194:49107 (local address 127.27.110.195:44927)
0812 07:06:24.617751 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.617756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.618042 (+ 286us) server_negotiation.cc:407] Connection header received
0812 07:06:24.618101 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.618103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.618170 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.618254 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 07:06:24.618502 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.619255 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.620733 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.621355 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.621487 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.621537 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.621827 (+ 290us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.621904 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260812 07:06:24.622051 31624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.617314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:43661 (local address 127.27.110.194:38665)
0812 07:06:24.617456 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.617473 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.617575 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.617909 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.617912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.617935 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:24.618075 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.618082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.619211 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.619214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.620569 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.620587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.621501 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.621505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.621681 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.621721 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.621788 (+ 67us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.621880 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":101}
I20260812 07:06:24.622061 31957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.617451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.195:44927 (local address 127.27.110.194:49107)
0812 07:06:24.617919 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.617932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:24.618022 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:24.618275 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:24.618277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:24.618293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 07:06:24.618386 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.618389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.619352 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.619355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.620569 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:24.620581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.621501 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.621505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:24.621642 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.621703 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 07:06:24.621788 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 07:06:24.621880 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":101,"thread_start_us":180,"threads_started":1}
I20260812 07:06:24.622813 31253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e9f10fc6a45a9a7c3baf23e934721" candidate_uuid: "8cc2252caeb046d6b5339c4fdb55806e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568404922b88405380d6925d581389c8" is_pre_election: true
I20260812 07:06:24.622821 31798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e9f10fc6a45a9a7c3baf23e934721" candidate_uuid: "8cc2252caeb046d6b5339c4fdb55806e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ceb566bbd14163a2965689beedfc3c" is_pre_election: true
I20260812 07:06:24.623180 31798 raft_consensus.cc:2454] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc2252caeb046d6b5339c4fdb55806e in term 0.
I20260812 07:06:24.623180 31253 raft_consensus.cc:2454] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cc2252caeb046d6b5339c4fdb55806e in term 0.
I20260812 07:06:24.623699 31374 leader_election.cc:302] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 568404922b88405380d6925d581389c8, 8cc2252caeb046d6b5339c4fdb55806e; no voters:
I20260812 07:06:24.623991 31953 raft_consensus.cc:2781] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 07:06:24.624078 31953 raft_consensus.cc:482] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 07:06:24.624109 31953 raft_consensus.cc:3037] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:24.625196 31953 raft_consensus.cc:504] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.625584 31953 leader_election.cc:288] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [CANDIDATE]: Term 1 election: Requested vote from peers 568404922b88405380d6925d581389c8 (127.27.110.193:43661), 18ceb566bbd14163a2965689beedfc3c (127.27.110.195:44927)
I20260812 07:06:24.625975 31798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e9f10fc6a45a9a7c3baf23e934721" candidate_uuid: "8cc2252caeb046d6b5339c4fdb55806e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18ceb566bbd14163a2965689beedfc3c"
I20260812 07:06:24.626067 31253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e9f10fc6a45a9a7c3baf23e934721" candidate_uuid: "8cc2252caeb046d6b5339c4fdb55806e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "568404922b88405380d6925d581389c8"
I20260812 07:06:24.626132 31798 raft_consensus.cc:3037] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:24.626243 31253 raft_consensus.cc:3037] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:24.627372 31798 raft_consensus.cc:2454] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc2252caeb046d6b5339c4fdb55806e in term 1.
I20260812 07:06:24.627424 31253 raft_consensus.cc:2454] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cc2252caeb046d6b5339c4fdb55806e in term 1.
I20260812 07:06:24.627732 31373 leader_election.cc:302] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18ceb566bbd14163a2965689beedfc3c, 8cc2252caeb046d6b5339c4fdb55806e; no voters:
I20260812 07:06:24.627980 31953 raft_consensus.cc:2781] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:24.628233 31953 raft_consensus.cc:686] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 1 LEADER]: Becoming Leader. State: Replica: 8cc2252caeb046d6b5339c4fdb55806e, State: Running, Role: LEADER
I20260812 07:06:24.628623 31953 consensus_queue.cc:237] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } }
I20260812 07:06:24.631932 30985 catalog_manager.cc:5495] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cc2252caeb046d6b5339c4fdb55806e (127.27.110.194). New cstate: current_term: 1 leader_uuid: "8cc2252caeb046d6b5339c4fdb55806e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cc2252caeb046d6b5339c4fdb55806e" member_type: VOTER last_known_addr { host: "127.27.110.194" port: 45997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 } health_report { overall_health: UNKNOWN } } }
I20260812 07:06:24.648604 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.644997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57868 (local address 127.27.110.194:45997)
0812 07:06:24.645141 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:24.645146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:24.645162 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:06:24.645261 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:24.645264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:24.645319 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:24.645423 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:24.645705 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.646213 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.647147 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:24.647368 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:24.647481 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:24.647793 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:24.648203 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:24.648341 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:24.648405 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 07:06:24.648459 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:06:24.654523 31798 raft_consensus.cc:1261] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 1 FOLLOWER]: Refusing update from remote peer 8cc2252caeb046d6b5339c4fdb55806e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:24.654531 31253 raft_consensus.cc:1261] T ff3e9f10fc6a45a9a7c3baf23e934721 P 568404922b88405380d6925d581389c8 [term 1 FOLLOWER]: Refusing update from remote peer 8cc2252caeb046d6b5339c4fdb55806e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:24.655331 31960 consensus_queue.cc:1035] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18ceb566bbd14163a2965689beedfc3c" member_type: VOTER last_known_addr { host: "127.27.110.195" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:24.655529 31953 consensus_queue.cc:1035] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [LEADER]: Connected to new peer: Peer: permanent_uuid: "568404922b88405380d6925d581389c8" member_type: VOTER last_known_addr { host: "127.27.110.193" port: 43661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:24.663308 31966 mvcc.cc:205] Tried to move back new op lower bound from 7317579303537745920 to 7317579303441645568. Current Snapshot: MvccSnapshot[applied={T|T < 7317579303537745920}]
I20260812 07:06:24.664019 31975 mvcc.cc:205] Tried to move back new op lower bound from 7317579303537745920 to 7317579303441645568. Current Snapshot: MvccSnapshot[applied={T|T < 7317579303537745920}]
I20260812 07:06:24.670275 30985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:24.670408 30985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 07:06:24.672798 30985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d8c6336d9e34824b04398c591f12cc8: Sending DeleteTablet for 3 replicas of tablet ff3e9f10fc6a45a9a7c3baf23e934721
I20260812 07:06:24.673683 31218 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3e9f10fc6a45a9a7c3baf23e934721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:24 UTC) from {username='slave'} at 127.0.0.1:48870
I20260812 07:06:24.673688 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3e9f10fc6a45a9a7c3baf23e934721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:24 UTC) from {username='slave'} at 127.0.0.1:57854
I20260812 07:06:24.673808 31754 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3e9f10fc6a45a9a7c3baf23e934721 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 07:06:24 UTC) from {username='slave'} at 127.0.0.1:36620
I20260812 07:06:24.674203 31992 tablet_replica.cc:330] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: stopping tablet replica
I20260812 07:06:24.674522 31992 raft_consensus.cc:2229] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 1 LEADER]: Raft consensus shutting down.
I20260812 07:06:24.674829 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 31104
I20260812 07:06:24.674911 31992 raft_consensus.cc:2258] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:24.686724 31992 ts_tablet_manager.cc:1905] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:24.691154 31992 ts_tablet_manager.cc:1918] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:24.691305 31992 log.cc:1196] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-1/wal/wals/ff3e9f10fc6a45a9a7c3baf23e934721
I20260812 07:06:24.691704 31992 ts_tablet_manager.cc:1939] T ff3e9f10fc6a45a9a7c3baf23e934721 P 8cc2252caeb046d6b5339c4fdb55806e: Deleting consensus metadata
I20260812 07:06:24.693001 31995 tablet_replica.cc:330] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: stopping tablet replica
I20260812 07:06:24.693287 30958 catalog_manager.cc:4841] TS 8cc2252caeb046d6b5339c4fdb55806e (127.27.110.194:45997): tablet ff3e9f10fc6a45a9a7c3baf23e934721 (table test-table [id=1a76dff520d649a881094fd0c66e7bf2]) successfully deleted
I20260812 07:06:24.693382 31995 raft_consensus.cc:2229] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 07:06:24.693661 31995 raft_consensus.cc:2258] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 07:06:24.695127 31995 ts_tablet_manager.cc:1905] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 07:06:24.697897 31995 ts_tablet_manager.cc:1918] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 07:06:24.698055 31995 log.cc:1196] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Deleting WAL directory at /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786518006756923-28091-0/minicluster-data/ts-2/wal/wals/ff3e9f10fc6a45a9a7c3baf23e934721
I20260812 07:06:24.698426 31995 ts_tablet_manager.cc:1939] T ff3e9f10fc6a45a9a7c3baf23e934721 P 18ceb566bbd14163a2965689beedfc3c: Deleting consensus metadata
W20260812 07:06:24.699610 30958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.110.193:43661 (error 108)
I20260812 07:06:24.699852 30957 catalog_manager.cc:4841] TS 18ceb566bbd14163a2965689beedfc3c (127.27.110.195:44927): tablet ff3e9f10fc6a45a9a7c3baf23e934721 (table test-table [id=1a76dff520d649a881094fd0c66e7bf2]) successfully deleted
I20260812 07:06:24.700567 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 31349
I20260812 07:06:24.700791 31928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:24.700456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.193:43661 (local address 127.0.0.1:48878)
0812 07:06:24.700606 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:24.700692 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.193:43661: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 07:06:24.706274 30958 catalog_manager.cc:4584] TS 568404922b88405380d6925d581389c8 (127.27.110.193:43661): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff3e9f10fc6a45a9a7c3baf23e934721: Network error: Client connection negotiation failed: client connection to 127.27.110.193:43661: connect: Connection refused (error 111)
I20260812 07:06:24.714859 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 31625
I20260812 07:06:24.727896 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 30926
2026-08-12T07:06:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1185 ms)
[----------] 6 tests from EncryptionPolicyTest (12141 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:24 dist-test-slave-w4v5 krb5kdc[32003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:24 dist-test-slave-w4v5 krb5kdc[32003](info): set up 2 sockets
Aug 12 07:06:24 dist-test-slave-w4v5 krb5kdc[32003](info): commencing operation
krb5kdc: starting...
W20260812 07:06:26.046562 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:26 dist-test-slave-w4v5 krb5kdc[32003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:26.177397 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:26.290519 32028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:26.290777 32028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:26.290817 32028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:26.294860 32028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:26.294924 32028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:26.294943 32028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:26.294960 32028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:26.294974 32028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:26.300168 32028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36191
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:26.301127 32028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:26.302103 32028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:26.310547 32040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:26.310722 32028 server_base.cc:1034] running on GCE node
W20260812 07:06:26.310580 32035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:26.310575 32036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:26.311445 32028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:26.312469 32028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:26.313664 32028 hybrid_clock.cc:648] HybridClock initialized: now 1786518386313650 us; error 37 us; skew 500 ppm
Aug 12 07:06:26 dist-test-slave-w4v5 krb5kdc[32003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:26.320724 32028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:26.321642 32028 webserver.cc:460] Webserver started at http://127.0.0.1:43869/ using document root <none> and password file <none>
I20260812 07:06:26.322208 32028 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:26.322300 32028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:26.322530 32028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:26.324635 32028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "92f88d4d60424f529a8b7ffe8eab25db"
format_stamp: "Formatted at 2026-08-12 07:06:26 on dist-test-slave-w4v5"
server_key: "4377b9761e32e6739caf83fc6f726ddb"
server_key_iv: "2afd5f1265d8ca3d3e674d30865a4edc"
server_key_version: "encryptionkey@0"
I20260812 07:06:26.325205 32028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "92f88d4d60424f529a8b7ffe8eab25db"
format_stamp: "Formatted at 2026-08-12 07:06:26 on dist-test-slave-w4v5"
server_key: "4377b9761e32e6739caf83fc6f726ddb"
server_key_iv: "2afd5f1265d8ca3d3e674d30865a4edc"
server_key_version: "encryptionkey@0"
I20260812 07:06:26.328835 32028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:06:26.331218 32053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:26.332377 32028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:26.332517 32028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "92f88d4d60424f529a8b7ffe8eab25db"
format_stamp: "Formatted at 2026-08-12 07:06:26 on dist-test-slave-w4v5"
server_key: "4377b9761e32e6739caf83fc6f726ddb"
server_key_iv: "2afd5f1265d8ca3d3e674d30865a4edc"
server_key_version: "encryptionkey@0"
I20260812 07:06:26.332621 32028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:26.348701 32028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:26.351886 32028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:26.352087 32028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:26.359454 32028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36191
I20260812 07:06:26.359472 32163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36191 every 8 connection(s)
I20260812 07:06:26.360380 32028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:26.362968 32166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:26.363968 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 32028
I20260812 07:06:26.364095 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:26.364382 28091 external_mini_cluster.cc:1428] Setting key 695d935c3418cc59b685a9d6455847f1
I20260812 07:06:26.369603 32166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db: Bootstrap starting.
I20260812 07:06:26.372213 32166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:26.372931 32166 log.cc:824] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:06:26 dist-test-slave-w4v5 krb5kdc[32003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:06:26.375125 32166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db: No bootstrap required, opened a new log
I20260812 07:06:26.379146 32166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f88d4d60424f529a8b7ffe8eab25db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36191 } }
I20260812 07:06:26.379385 32166 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:26.379457 32166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92f88d4d60424f529a8b7ffe8eab25db, State: Initialized, Role: FOLLOWER
I20260812 07:06:26.379614 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:26.365798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38827 (local address 127.0.0.1:36191)
0812 07:06:26.366193 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:26.366216 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:26.366262 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:06:26.367111 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:26.367135 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:26.367551 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:26.367927 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:26.368682 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:26.370119 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:26.371029 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:26.371734 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:26.371893 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:26.374606 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:26.374611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:26.374625 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:26.374651 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:26.376432 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:26.377042 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:26.377046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:26.377051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:26.377159 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:26.377514 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:26.377516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:26.377517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:26.377898 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:26.378113 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:26.378444 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 07:06:26.378705 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":99,"threads_started":1}
I20260812 07:06:26.379961 32166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f88d4d60424f529a8b7ffe8eab25db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36191 } }
I20260812 07:06:26.380119 32166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:26.380193 32166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:26.380288 32166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:26.381237 32166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f88d4d60424f529a8b7ffe8eab25db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36191 } }
I20260812 07:06:26.381601 32166 leader_election.cc:302] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92f88d4d60424f529a8b7ffe8eab25db; no voters:
I20260812 07:06:26.381911 32166 leader_election.cc:288] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:26.382050 32175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:26.382279 32175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [term 1 LEADER]: Becoming Leader. State: Replica: 92f88d4d60424f529a8b7ffe8eab25db, State: Running, Role: LEADER
I20260812 07:06:26.382591 32175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f88d4d60424f529a8b7ffe8eab25db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36191 } }
I20260812 07:06:26.382726 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:26.383219 32166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:26.384382 32179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92f88d4d60424f529a8b7ffe8eab25db. Latest consensus state: current_term: 1 leader_uuid: "92f88d4d60424f529a8b7ffe8eab25db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f88d4d60424f529a8b7ffe8eab25db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36191 } } }
I20260812 07:06:26.384512 32179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:26.384864 32178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92f88d4d60424f529a8b7ffe8eab25db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92f88d4d60424f529a8b7ffe8eab25db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36191 } } }
I20260812 07:06:26.384969 32178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db [sys.catalog]: This master's current role is: LEADER
W20260812 07:06:26.388001 32212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:26.388089 32212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:26.388199 32199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:26.389065 32199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:26.395123 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:26.386221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59601 (local address 127.0.0.1:36191)
0812 07:06:26.386343 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:26.386347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:26.386605 (+ 258us) server_negotiation.cc:407] Connection header received
0812 07:06:26.386715 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:26.386720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:26.386771 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:26.386857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:26.387681 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:26.388426 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:26.389544 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:26.389912 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:26.390017 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:26.391864 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:26.391868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:26.391871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:26.391902 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:26.393316 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:26.393792 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:26.393801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:26.393803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:26.393875 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:26.394205 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:26.394208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:26.394210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:26.394377 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:26.394532 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:26.394843 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 07:06:26.394936 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260812 07:06:26.396188 32199 catalog_manager.cc:1347] Generated new cluster ID: 58694d5cb40e49abbedcc2f3277fa660
I20260812 07:06:26.396246 32199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:26.418730 32199 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:26.419735 32199 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:26.426716 32199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92f88d4d60424f529a8b7ffe8eab25db: Generated new TSK 0
I20260812 07:06:26.427426 32199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:26.433650 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 32028
W20260812 07:06:26.464389 32000 connection.cc:535] client connection to 127.0.0.1:36191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36191): Connection reset by peer (error 104)
2026-08-12T07:06:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1723 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:26 dist-test-slave-w4v5 krb5kdc[32235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:26 dist-test-slave-w4v5 krb5kdc[32235](info): set up 2 sockets
Aug 12 07:06:26 dist-test-slave-w4v5 krb5kdc[32235](info): commencing operation
krb5kdc: starting...
W20260812 07:06:27.764988 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:27 dist-test-slave-w4v5 krb5kdc[32235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:27.895462 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:28.008339 32258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:28.008646 32258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:28.008725 32258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:28.012720 32258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:28.012812 32258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:28.012850 32258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:28.012887 32258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:28.012921 32258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:28.018162 32258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:28.019284 32258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:28.020323 32258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:28.029084 32277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:28.029284 32258 server_base.cc:1034] running on GCE node
W20260812 07:06:28.029176 32270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:28.029165 32267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:28.029981 32258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:28.031157 32258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:28.032343 32258 hybrid_clock.cc:648] HybridClock initialized: now 1786518388032322 us; error 38 us; skew 500 ppm
Aug 12 07:06:28 dist-test-slave-w4v5 krb5kdc[32235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:28.035250 32258 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:28.036108 32258 webserver.cc:460] Webserver started at http://127.0.0.1:45265/ using document root <none> and password file <none>
I20260812 07:06:28.036635 32258 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:28.036695 32258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:28.036863 32258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:28.038906 32258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "a52497262b5d41bc9d0c21b5bd7a7a37"
format_stamp: "Formatted at 2026-08-12 07:06:28 on dist-test-slave-w4v5"
server_key: "3dc4fddd95abe88b9409f8fdaf7b7e74"
server_key_iv: "b7aa6551c0eeb3dafd8a899c6f2ef270"
server_key_version: "encryptionkey@0"
I20260812 07:06:28.039527 32258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "a52497262b5d41bc9d0c21b5bd7a7a37"
format_stamp: "Formatted at 2026-08-12 07:06:28 on dist-test-slave-w4v5"
server_key: "3dc4fddd95abe88b9409f8fdaf7b7e74"
server_key_iv: "b7aa6551c0eeb3dafd8a899c6f2ef270"
server_key_version: "encryptionkey@0"
I20260812 07:06:28.043764 32258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260812 07:06:28.046113 32286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:28.047533 32258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260812 07:06:28.047657 32258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "a52497262b5d41bc9d0c21b5bd7a7a37"
format_stamp: "Formatted at 2026-08-12 07:06:28 on dist-test-slave-w4v5"
server_key: "3dc4fddd95abe88b9409f8fdaf7b7e74"
server_key_iv: "b7aa6551c0eeb3dafd8a899c6f2ef270"
server_key_version: "encryptionkey@0"
I20260812 07:06:28.047753 32258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:28.058094 32258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:28.060848 32258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:28.060997 32258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:28.068003 32258 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36483
I20260812 07:06:28.068064 32412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36483 every 8 connection(s)
I20260812 07:06:28.069034 32258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:28.071790 32413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:28.072088 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 32258
I20260812 07:06:28.072237 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:28.072566 28091 external_mini_cluster.cc:1428] Setting key 17eed7f7bf81c2a1be23d2d78551545e
I20260812 07:06:28.078356 32413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37: Bootstrap starting.
I20260812 07:06:28.080787 32413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:28.081640 32413 log.cc:824] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:28.081709 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:28.073979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56711 (local address 127.0.0.1:36483)
0812 07:06:28.074414 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:28.074429 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:28.074464 (+ 35us) server_negotiation.cc:407] Connection header received
0812 07:06:28.075244 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:28.075264 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:28.075600 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:28.075933 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:28.076345 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:28.077325 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:28.078287 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:28.078850 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:28.079016 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:28.079991 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:28.079996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:28.080009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:28.080393 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:28.080445 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:28.080468 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:28.080607 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:28.080787 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 07:06:28.080933 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":139,"threads_started":1}
I20260812 07:06:28.083833 32413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37: No bootstrap required, opened a new log
I20260812 07:06:28.084640 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:28.088009 32413 raft_consensus.cc:348] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260812 07:06:28.088279 32413 raft_consensus.cc:374] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:28.088340 32413 raft_consensus.cc:729] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52497262b5d41bc9d0c21b5bd7a7a37, State: Initialized, Role: FOLLOWER
I20260812 07:06:28.088807 32413 consensus_queue.cc:260] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260812 07:06:28.088975 32413 raft_consensus.cc:388] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:28.089042 32413 raft_consensus.cc:482] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:28.089143 32413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:28.090294 32413 raft_consensus.cc:504] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260812 07:06:28.090705 32413 leader_election.cc:302] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a52497262b5d41bc9d0c21b5bd7a7a37; no voters:
I20260812 07:06:28.090968 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:28.086642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46725 (local address 127.0.0.1:36483)
0812 07:06:28.086767 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:28.086771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:28.087002 (+ 231us) server_negotiation.cc:407] Connection header received
0812 07:06:28.087161 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:28.087164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:28.087208 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:28.087276 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:28.087577 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:28.088085 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:28.089264 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:28.089511 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:28.089591 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:28.090494 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:28.090497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:28.090499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:28.090510 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:28.090533 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:28.090543 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:28.090611 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:28.090783 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 07:06:28.090811 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:06:28.091080 32413 leader_election.cc:288] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:28.091261 32430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:28.091461 32430 raft_consensus.cc:686] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [term 1 LEADER]: Becoming Leader. State: Replica: a52497262b5d41bc9d0c21b5bd7a7a37, State: Running, Role: LEADER
I20260812 07:06:28.091813 32430 consensus_queue.cc:237] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260812 07:06:28.092391 32413 sys_catalog.cc:564] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:28.092998 32430 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a52497262b5d41bc9d0c21b5bd7a7a37. Latest consensus state: current_term: 1 leader_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } } }
I20260812 07:06:28.093211 32430 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:28.093886 32435 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52497262b5d41bc9d0c21b5bd7a7a37" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } } }
I20260812 07:06:28.094040 32435 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:28.094478 32456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:28.097215 32456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:28.102563 32456 catalog_manager.cc:1347] Generated new cluster ID: 2ef23dbaa91847788271eb0978e63a59
I20260812 07:06:28.102627 32456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:28.139981 32456 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:28.141316 32456 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:28.155128 32456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a52497262b5d41bc9d0c21b5bd7a7a37: Generated new TSK 0
I20260812 07:06:28.155988 32456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:28.161152 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 32258
W20260812 07:06:28.177263 32231 connection.cc:535] client connection to 127.0.0.1:36483 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36483): Connection reset by peer (error 104)
2026-08-12T07:06:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1712 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:28 dist-test-slave-w4v5 krb5kdc[32481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:28 dist-test-slave-w4v5 krb5kdc[32481](info): set up 2 sockets
Aug 12 07:06:28 dist-test-slave-w4v5 krb5kdc[32481](info): commencing operation
krb5kdc: starting...
W20260812 07:06:29.472247 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:29 dist-test-slave-w4v5 krb5kdc[32481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:29.602010 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:29.714835 32515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:29.715180 32515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:29.715261 32515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:29.719206 32515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:29.719300 32515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:29.719340 32515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:29.719378 32515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:29.719416 32515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:29.724642 32515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:29.725752 32515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:29.726823 32515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:29.735641 32535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:29.735625 32529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:29.735733 32539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:29.736063 32515 server_base.cc:1034] running on GCE node
I20260812 07:06:29.736712 32515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:29.737924 32515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:29.739161 32515 hybrid_clock.cc:648] HybridClock initialized: now 1786518389739140 us; error 27 us; skew 500 ppm
Aug 12 07:06:29 dist-test-slave-w4v5 krb5kdc[32481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:29.742184 32515 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:29.743166 32515 webserver.cc:460] Webserver started at http://127.0.0.1:35363/ using document root <none> and password file <none>
I20260812 07:06:29.743731 32515 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:29.743819 32515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:29.744061 32515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:29.746189 32515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "b64558262cf04f25b3dc8ee361fc1424"
format_stamp: "Formatted at 2026-08-12 07:06:29 on dist-test-slave-w4v5"
server_key: "0b60cd1725e728ad76a659c0973fd865"
server_key_iv: "707c0fd6e9f5e19e3e9b7e4432f14f92"
server_key_version: "encryptionkey@0"
I20260812 07:06:29.746757 32515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "b64558262cf04f25b3dc8ee361fc1424"
format_stamp: "Formatted at 2026-08-12 07:06:29 on dist-test-slave-w4v5"
server_key: "0b60cd1725e728ad76a659c0973fd865"
server_key_iv: "707c0fd6e9f5e19e3e9b7e4432f14f92"
server_key_version: "encryptionkey@0"
I20260812 07:06:29.750567 32515 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 07:06:29.753067 32554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:29.754297 32515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:06:29.754454 32515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "b64558262cf04f25b3dc8ee361fc1424"
format_stamp: "Formatted at 2026-08-12 07:06:29 on dist-test-slave-w4v5"
server_key: "0b60cd1725e728ad76a659c0973fd865"
server_key_iv: "707c0fd6e9f5e19e3e9b7e4432f14f92"
server_key_version: "encryptionkey@0"
I20260812 07:06:29.754571 32515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:29.759548 32515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:29.762938 32515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:29.763192 32515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:29.770460 32515 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45061
I20260812 07:06:29.770468 32648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45061 every 8 connection(s)
I20260812 07:06:29.771479 32515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:29.774555 32649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:29.778579 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 32515
I20260812 07:06:29.778738 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:29.779062 28091 external_mini_cluster.cc:1428] Setting key 214ae73d0fcd02875c8c73eabd15f24f
I20260812 07:06:29.780759 32649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424: Bootstrap starting.
I20260812 07:06:29.783320 32649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:29.784122 32649 log.cc:824] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:29.784418 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:29.780506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47283 (local address 127.0.0.1:45061)
0812 07:06:29.780935 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:29.780950 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:29.780986 (+ 36us) server_negotiation.cc:407] Connection header received
0812 07:06:29.781602 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:29.781627 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:29.781909 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:29.782194 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:29.783053 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:29.783057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:29.783070 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:29.783122 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:29.783184 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:29.783209 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:29.783296 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:29.783483 (+ 187us) server_negotiation.cc:299] Negotiation successful
0812 07:06:29.783701 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":140,"threads_started":1}
I20260812 07:06:29.786051 32649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424: No bootstrap required, opened a new log
I20260812 07:06:29.787173 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:29.791127 32653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:29.789480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56369 (local address 127.0.0.1:45061)
0812 07:06:29.789616 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:29.789621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:29.789752 (+ 131us) server_negotiation.cc:407] Connection header received
0812 07:06:29.789864 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:29.789870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:29.789905 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:29.789990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:29.790737 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:29.790741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:29.790744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:29.790756 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:29.790783 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:29.790795 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:29.790884 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:29.790947 (+ 63us) server_negotiation.cc:299] Negotiation successful
0812 07:06:29.790988 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:06:29.790937 32649 raft_consensus.cc:348] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64558262cf04f25b3dc8ee361fc1424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260812 07:06:29.791240 32649 raft_consensus.cc:374] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:29.791313 32649 raft_consensus.cc:729] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64558262cf04f25b3dc8ee361fc1424, State: Initialized, Role: FOLLOWER
I20260812 07:06:29.791795 32649 consensus_queue.cc:260] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64558262cf04f25b3dc8ee361fc1424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260812 07:06:29.791966 32649 raft_consensus.cc:388] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:29.792058 32649 raft_consensus.cc:482] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:29.792166 32649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:29.793095 32649 raft_consensus.cc:504] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64558262cf04f25b3dc8ee361fc1424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260812 07:06:29.793435 32649 leader_election.cc:302] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b64558262cf04f25b3dc8ee361fc1424; no voters:
I20260812 07:06:29.793757 32649 leader_election.cc:288] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:29.793838 32678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:29.794061 32678 raft_consensus.cc:686] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [term 1 LEADER]: Becoming Leader. State: Replica: b64558262cf04f25b3dc8ee361fc1424, State: Running, Role: LEADER
I20260812 07:06:29.794404 32678 consensus_queue.cc:237] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64558262cf04f25b3dc8ee361fc1424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } }
I20260812 07:06:29.794951 32649 sys_catalog.cc:564] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:29.796059 32680 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b64558262cf04f25b3dc8ee361fc1424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64558262cf04f25b3dc8ee361fc1424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } } }
I20260812 07:06:29.796198 32680 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:29.796581 32700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:29.796563 32683 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64558262cf04f25b3dc8ee361fc1424. Latest consensus state: current_term: 1 leader_uuid: "b64558262cf04f25b3dc8ee361fc1424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b64558262cf04f25b3dc8ee361fc1424" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45061 } } }
I20260812 07:06:29.797454 32683 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:29.799889 32700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:29.805511 32700 catalog_manager.cc:1347] Generated new cluster ID: 2888e830711148ecaab64265ec9f2a7c
I20260812 07:06:29.805593 32700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:29.827481 32700 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:29.827860 32700 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:29.834508 32700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b64558262cf04f25b3dc8ee361fc1424: Generated new TSK 0
I20260812 07:06:29.835280 32700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:29.838963 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 32515
2026-08-12T07:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1679 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:29 dist-test-slave-w4v5 krb5kdc[32723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:29 dist-test-slave-w4v5 krb5kdc[32723](info): set up 2 sockets
Aug 12 07:06:29 dist-test-slave-w4v5 krb5kdc[32723](info): commencing operation
krb5kdc: starting...
W20260812 07:06:31.158388 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.287s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:31 dist-test-slave-w4v5 krb5kdc[32723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:31.289893 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:31.401048 32744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:31.401335 32744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:31.401404 32744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:31.405318 32744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:31.405405 32744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:31.405443 32744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:31.405495 32744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:31.405530 32744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:31.410681 32744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:31.411736 32744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:31.412739 32744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:31.421212 32760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:31.421213 32757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:31.421250 32758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:31.421492 32744 server_base.cc:1034] running on GCE node
I20260812 07:06:31.422232 32744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:31.423381 32744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:31.424567 32744 hybrid_clock.cc:648] HybridClock initialized: now 1786518391424543 us; error 42 us; skew 500 ppm
Aug 12 07:06:31 dist-test-slave-w4v5 krb5kdc[32723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518391, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:31.430086 32744 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:31.431018 32744 webserver.cc:460] Webserver started at http://127.0.0.1:43609/ using document root <none> and password file <none>
I20260812 07:06:31.431599 32744 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:31.431689 32744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:31.431911 32744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:31.433959 32744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "257a371084214f68b2375983a168169e"
format_stamp: "Formatted at 2026-08-12 07:06:31 on dist-test-slave-w4v5"
server_key: "97830b7d14630a0261cd98e37c432c17"
server_key_iv: "be5579722784dd516233d51cecc8c72a"
server_key_version: "encryptionkey@0"
I20260812 07:06:31.434535 32744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "257a371084214f68b2375983a168169e"
format_stamp: "Formatted at 2026-08-12 07:06:31 on dist-test-slave-w4v5"
server_key: "97830b7d14630a0261cd98e37c432c17"
server_key_iv: "be5579722784dd516233d51cecc8c72a"
server_key_version: "encryptionkey@0"
I20260812 07:06:31.438261 32744 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:06:31.440757 301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:31.442000 32744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:31.442147 32744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "257a371084214f68b2375983a168169e"
format_stamp: "Formatted at 2026-08-12 07:06:31 on dist-test-slave-w4v5"
server_key: "97830b7d14630a0261cd98e37c432c17"
server_key_iv: "be5579722784dd516233d51cecc8c72a"
server_key_version: "encryptionkey@0"
I20260812 07:06:31.442250 32744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:31.464478 32744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:31.467594 32744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:31.467793 32744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:31.475621 32744 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35837
I20260812 07:06:31.475649 418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35837 every 8 connection(s)
I20260812 07:06:31.476550 32744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:31.479343 420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:31.484817 420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e: Bootstrap starting.
I20260812 07:06:31.486805 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 32744
I20260812 07:06:31.486938 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:31.487286 28091 external_mini_cluster.cc:1428] Setting key bda921573e4920284be7b2c95669063d
I20260812 07:06:31.487552 420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:31.488334 420 log.cc:824] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:31.490365 420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e: No bootstrap required, opened a new log
I20260812 07:06:31.493748 420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a371084214f68b2375983a168169e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260812 07:06:31.493992 420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:31.494123 420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 257a371084214f68b2375983a168169e, State: Initialized, Role: FOLLOWER
I20260812 07:06:31.494613 420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a371084214f68b2375983a168169e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260812 07:06:31.494745 420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:31.494835 420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:31.494948 420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:31.496026 420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a371084214f68b2375983a168169e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
Aug 12 07:06:31 dist-test-slave-w4v5 krb5kdc[32723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:06:31.496491 420 leader_election.cc:302] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 257a371084214f68b2375983a168169e; no voters:
I20260812 07:06:31.496884 420 leader_election.cc:288] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:31.496973 430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:31.497241 430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [term 1 LEADER]: Becoming Leader. State: Replica: 257a371084214f68b2375983a168169e, State: Running, Role: LEADER
I20260812 07:06:31.497589 430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a371084214f68b2375983a168169e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } }
I20260812 07:06:31.498268 420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:31.499292 433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 257a371084214f68b2375983a168169e. Latest consensus state: current_term: 1 leader_uuid: "257a371084214f68b2375983a168169e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a371084214f68b2375983a168169e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } } }
I20260812 07:06:31.499428 433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:31.499742 431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "257a371084214f68b2375983a168169e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "257a371084214f68b2375983a168169e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35837 } } }
I20260812 07:06:31.499847 431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:31.502408 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:31.488467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46917 (local address 127.0.0.1:35837)
0812 07:06:31.489021 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:31.489043 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:31.489090 (+ 47us) server_negotiation.cc:407] Connection header received
0812 07:06:31.489786 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:31.489804 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:31.490121 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:31.490512 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:31.491372 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:31.492363 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:31.493239 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:31.493705 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:31.493901 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:31.496845 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:31.496852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:31.496870 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:31.496904 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:31.499035 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:31.499655 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:31.499664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:31.499670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:31.499830 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:31.500154 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:31.500161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:31.500166 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:31.500678 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:31.500886 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:31.501197 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 07:06:31.501519 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"spinlock_wait_cycles":25984,"thread_start_us":144,"threads_started":1}
W20260812 07:06:31.503608 464 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:31.503687 464 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:31.503769 445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:31.504624 445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:31.505275 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:31.510540 445 catalog_manager.cc:1347] Generated new cluster ID: a038ff4f82944e42b0c1a3806120b1fc
I20260812 07:06:31.510612 445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:31.516577 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:31.507267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48381 (local address 127.0.0.1:35837)
0812 07:06:31.507385 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:31.507389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:31.507635 (+ 246us) server_negotiation.cc:407] Connection header received
0812 07:06:31.507756 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:31.507759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:31.507804 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:31.507886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:31.508695 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:31.509864 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:31.511059 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:31.511446 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:31.511542 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:31.513222 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:31.513226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:31.513229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:31.513252 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:31.514885 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:31.515392 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:31.515396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:31.515397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:31.515476 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:31.515758 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:31.515761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:31.515763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:31.515948 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:31.516069 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:31.516323 (+ 254us) server_negotiation.cc:299] Negotiation successful
0812 07:06:31.516429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260812 07:06:31.524766 445 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:31.525696 445 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:31.532895 445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 257a371084214f68b2375983a168169e: Generated new TSK 0
I20260812 07:06:31.533689 445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:31.540468 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 32744
2026-08-12T07:06:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1703 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:31 dist-test-slave-w4v5 krb5kdc[495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:31 dist-test-slave-w4v5 krb5kdc[495](info): set up 2 sockets
Aug 12 07:06:31 dist-test-slave-w4v5 krb5kdc[495](info): commencing operation
krb5kdc: starting...
W20260812 07:06:32.865094 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.291s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:32 dist-test-slave-w4v5 krb5kdc[495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:32.989903 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:33.099817 514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:33.100106 514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:33.100179 514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:33.104110 514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:33.104202 514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:33.104238 514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:33.104271 514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:33.104307 514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:33.109406 514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:33.110478 514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:33.111526 514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:33.120071 528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:33.120024 535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:33.120045 529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:33.120273 514 server_base.cc:1034] running on GCE node
I20260812 07:06:33.120931 514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:33.121994 514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:33.123167 514 hybrid_clock.cc:648] HybridClock initialized: now 1786518393123138 us; error 40 us; skew 500 ppm
Aug 12 07:06:33 dist-test-slave-w4v5 krb5kdc[495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:33.125910 514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:33.126806 514 webserver.cc:460] Webserver started at http://127.0.0.1:46543/ using document root <none> and password file <none>
I20260812 07:06:33.127411 514 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:33.127476 514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:33.127707 514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:33.129702 514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "eca9def0f6ce48fba62fdf281f4e3953"
format_stamp: "Formatted at 2026-08-12 07:06:33 on dist-test-slave-w4v5"
server_key: "fac18eb69a67a4f8c33672ab6866a531"
server_key_iv: "c9c7a19708b15165bf3e191ffc26fbd3"
server_key_version: "encryptionkey@0"
I20260812 07:06:33.130244 514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "eca9def0f6ce48fba62fdf281f4e3953"
format_stamp: "Formatted at 2026-08-12 07:06:33 on dist-test-slave-w4v5"
server_key: "fac18eb69a67a4f8c33672ab6866a531"
server_key_iv: "c9c7a19708b15165bf3e191ffc26fbd3"
server_key_version: "encryptionkey@0"
I20260812 07:06:33.133893 514 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260812 07:06:33.136214 554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:33.137401 514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:33.137537 514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "eca9def0f6ce48fba62fdf281f4e3953"
format_stamp: "Formatted at 2026-08-12 07:06:33 on dist-test-slave-w4v5"
server_key: "fac18eb69a67a4f8c33672ab6866a531"
server_key_iv: "c9c7a19708b15165bf3e191ffc26fbd3"
server_key_version: "encryptionkey@0"
I20260812 07:06:33.137639 514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:33.154649 514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:33.157371 514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:33.157564 514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:33.164577 514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44575
I20260812 07:06:33.164618 649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44575 every 8 connection(s)
I20260812 07:06:33.165494 514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:33.165745 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 514
I20260812 07:06:33.165899 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:33.166255 28091 external_mini_cluster.cc:1428] Setting key d0eba49cb04d8ed2e91c5881424c8f1b
I20260812 07:06:33.168766 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:33.175635 653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953: Bootstrap starting.
I20260812 07:06:33.176342 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:33.167569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56131 (local address 127.0.0.1:44575)
0812 07:06:33.167984 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:33.168003 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:33.168043 (+ 40us) server_negotiation.cc:407] Connection header received
0812 07:06:33.168890 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:33.168912 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:33.169327 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:33.169674 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:33.170069 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:33.171605 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:33.172630 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:33.173307 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:33.173483 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:33.174327 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:33.174334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:33.174350 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:33.174858 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:33.174926 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:33.174957 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:33.175055 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:33.175233 (+ 178us) server_negotiation.cc:299] Negotiation successful
0812 07:06:33.175441 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260812 07:06:33.178192 653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:33.179051 653 log.cc:824] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:33.179270 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:33.181102 653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953: No bootstrap required, opened a new log
I20260812 07:06:33.185328 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:33.181362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48725 (local address 127.0.0.1:44575)
0812 07:06:33.181466 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:33.181470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:33.181645 (+ 175us) server_negotiation.cc:407] Connection header received
0812 07:06:33.181738 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:33.181741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:33.181785 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:33.181866 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:33.182100 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:33.182833 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:33.183671 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:33.184082 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:33.184194 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:33.184942 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:33.184949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:33.184953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:33.184967 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:33.184994 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:33.185007 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:33.185081 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:33.185145 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 07:06:33.185199 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 07:06:33.185802 653 raft_consensus.cc:348] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca9def0f6ce48fba62fdf281f4e3953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260812 07:06:33.186013 653 raft_consensus.cc:374] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:33.186069 653 raft_consensus.cc:729] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eca9def0f6ce48fba62fdf281f4e3953, State: Initialized, Role: FOLLOWER
I20260812 07:06:33.186565 653 consensus_queue.cc:260] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca9def0f6ce48fba62fdf281f4e3953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260812 07:06:33.186717 653 raft_consensus.cc:388] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:33.186784 653 raft_consensus.cc:482] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:33.186880 653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:33.187835 653 raft_consensus.cc:504] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca9def0f6ce48fba62fdf281f4e3953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260812 07:06:33.188192 653 leader_election.cc:302] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eca9def0f6ce48fba62fdf281f4e3953; no voters:
I20260812 07:06:33.188544 653 leader_election.cc:288] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:33.189051 669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:33.189347 669 raft_consensus.cc:686] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [term 1 LEADER]: Becoming Leader. State: Replica: eca9def0f6ce48fba62fdf281f4e3953, State: Running, Role: LEADER
I20260812 07:06:33.189800 669 consensus_queue.cc:237] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca9def0f6ce48fba62fdf281f4e3953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } }
I20260812 07:06:33.189898 653 sys_catalog.cc:564] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:33.191771 670 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eca9def0f6ce48fba62fdf281f4e3953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca9def0f6ce48fba62fdf281f4e3953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } } }
I20260812 07:06:33.191946 670 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:33.191771 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eca9def0f6ce48fba62fdf281f4e3953. Latest consensus state: current_term: 1 leader_uuid: "eca9def0f6ce48fba62fdf281f4e3953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eca9def0f6ce48fba62fdf281f4e3953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44575 } } }
I20260812 07:06:33.192528 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:33.192689 696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:33.196156 696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:33.201784 696 catalog_manager.cc:1347] Generated new cluster ID: 38166edd3fad48e09114a46e28b8f8db
I20260812 07:06:33.201864 696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:33.218743 696 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:33.220031 696 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:33.226336 696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eca9def0f6ce48fba62fdf281f4e3953: Generated new TSK 0
I20260812 07:06:33.227010 696 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:33.232748 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 514
2026-08-12T07:06:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1690 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:33 dist-test-slave-w4v5 krb5kdc[712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:33 dist-test-slave-w4v5 krb5kdc[712](info): set up 2 sockets
Aug 12 07:06:33 dist-test-slave-w4v5 krb5kdc[712](info): commencing operation
krb5kdc: starting...
W20260812 07:06:34.533797 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:34 dist-test-slave-w4v5 krb5kdc[712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:34.662533 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:34.776243 740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:34.776528 740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:34.776625 740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:34.780720 740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:34.780799 740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:34.780846 740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:34.780887 740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:34.780930 740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:34.786124 740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:34.787164 740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:34.788230 740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:34.796720 752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:34.796867 740 server_base.cc:1034] running on GCE node
W20260812 07:06:34.796753 754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:34.796787 751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:34.797551 740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:34.798686 740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:34.799865 740 hybrid_clock.cc:648] HybridClock initialized: now 1786518394799838 us; error 42 us; skew 500 ppm
Aug 12 07:06:34 dist-test-slave-w4v5 krb5kdc[712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:34.802644 740 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:34.803544 740 webserver.cc:460] Webserver started at http://127.0.0.1:45119/ using document root <none> and password file <none>
I20260812 07:06:34.804050 740 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:34.804117 740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:34.804279 740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:34.806195 740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "0223275e39154268b58a501674076e45"
format_stamp: "Formatted at 2026-08-12 07:06:34 on dist-test-slave-w4v5"
server_key: "9f21cf1103f94dd86fd8a32a08ffaf77"
server_key_iv: "88d274741854c2b0163f9118a521a42d"
server_key_version: "encryptionkey@0"
I20260812 07:06:34.806701 740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "0223275e39154268b58a501674076e45"
format_stamp: "Formatted at 2026-08-12 07:06:34 on dist-test-slave-w4v5"
server_key: "9f21cf1103f94dd86fd8a32a08ffaf77"
server_key_iv: "88d274741854c2b0163f9118a521a42d"
server_key_version: "encryptionkey@0"
I20260812 07:06:34.810420 740 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:06:34.812830 781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:34.814185 740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:34.814296 740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "0223275e39154268b58a501674076e45"
format_stamp: "Formatted at 2026-08-12 07:06:34 on dist-test-slave-w4v5"
server_key: "9f21cf1103f94dd86fd8a32a08ffaf77"
server_key_iv: "88d274741854c2b0163f9118a521a42d"
server_key_version: "encryptionkey@0"
I20260812 07:06:34.814386 740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:34.818995 740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:34.822224 740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:34.822371 740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:34.829782 740 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38571
I20260812 07:06:34.829806 879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38571 every 8 connection(s)
I20260812 07:06:34.830730 740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:34.833442 880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:34.838820 880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45: Bootstrap starting.
I20260812 07:06:34.839128 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 740
I20260812 07:06:34.839280 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:34.839558 28091 external_mini_cluster.cc:1428] Setting key b50be53b29d367f245f2890022d5855d
I20260812 07:06:34.841538 880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:34.842444 880 log.cc:824] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:34.844797 880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45: No bootstrap required, opened a new log
I20260812 07:06:34.845323 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:34.840954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36185 (local address 127.0.0.1:38571)
0812 07:06:34.841423 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:34.841445 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:34.841491 (+ 46us) server_negotiation.cc:407] Connection header received
0812 07:06:34.842366 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:34.842389 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:34.842748 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:34.843137 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:34.843870 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:34.843876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:34.843889 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:34.843901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:34.843966 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:34.844000 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:34.844123 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:34.844258 (+ 135us) server_negotiation.cc:299] Negotiation successful
0812 07:06:34.844409 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260812 07:06:34.848340 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:34.849107 880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0223275e39154268b58a501674076e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260812 07:06:34.849325 880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:34.849392 880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0223275e39154268b58a501674076e45, State: Initialized, Role: FOLLOWER
I20260812 07:06:34.849861 880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0223275e39154268b58a501674076e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260812 07:06:34.850000 880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:34.850066 880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:34.850188 880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:34.851459 880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0223275e39154268b58a501674076e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260812 07:06:34.851856 880 leader_election.cc:302] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0223275e39154268b58a501674076e45; no voters:
I20260812 07:06:34.852226 880 leader_election.cc:288] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:34.852289 903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:34.852502 903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [term 1 LEADER]: Becoming Leader. State: Replica: 0223275e39154268b58a501674076e45, State: Running, Role: LEADER
I20260812 07:06:34.852560 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:34.850644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36047 (local address 127.0.0.1:38571)
0812 07:06:34.850770 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:34.850776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:34.850936 (+ 160us) server_negotiation.cc:407] Connection header received
0812 07:06:34.851065 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:34.851067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:34.851138 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:34.851214 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:34.852090 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:34.852094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:34.852096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:34.852113 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:34.852142 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:34.852152 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:34.852256 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:34.852372 (+ 116us) server_negotiation.cc:299] Negotiation successful
0812 07:06:34.852410 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 07:06:34.852809 903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0223275e39154268b58a501674076e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } }
I20260812 07:06:34.853546 880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:34.854046 903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0223275e39154268b58a501674076e45. Latest consensus state: current_term: 1 leader_uuid: "0223275e39154268b58a501674076e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0223275e39154268b58a501674076e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260812 07:06:34.854159 903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:34.854369 903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0223275e39154268b58a501674076e45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0223275e39154268b58a501674076e45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38571 } } }
I20260812 07:06:34.854478 903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:34.855882 913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:34.858947 913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:34.864320 913 catalog_manager.cc:1347] Generated new cluster ID: 65d4f057f6ce452787a58eac93625a90
I20260812 07:06:34.864392 913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:34.888139 913 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:34.888492 913 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:34.894032 913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0223275e39154268b58a501674076e45: Generated new TSK 0
I20260812 07:06:34.894788 913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:34.907934 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 740
2026-08-12T07:06:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1674 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:34 dist-test-slave-w4v5 krb5kdc[949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:34 dist-test-slave-w4v5 krb5kdc[949](info): set up 2 sockets
Aug 12 07:06:34 dist-test-slave-w4v5 krb5kdc[949](info): commencing operation
krb5kdc: starting...
W20260812 07:06:36.218286 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:36 dist-test-slave-w4v5 krb5kdc[949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:36.350050 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:36.462917 976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:36.463266 976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:36.463356 976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:36.467913 976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:36.468003 976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:36.468040 976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:36.468081 976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:36.468114 976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:36.473306 976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:36.474339 976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:36.475374 976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:36.483613 997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:36.483619 990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:36.483680 989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:36.483847 976 server_base.cc:1034] running on GCE node
I20260812 07:06:36.484604 976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:36.485592 976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:36.486754 976 hybrid_clock.cc:648] HybridClock initialized: now 1786518396486736 us; error 29 us; skew 500 ppm
Aug 12 07:06:36 dist-test-slave-w4v5 krb5kdc[949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:36.494397 976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:36.495447 976 webserver.cc:460] Webserver started at http://127.0.0.1:45019/ using document root <none> and password file <none>
I20260812 07:06:36.496017 976 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:36.496088 976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:36.496308 976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:36.498399 976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "1c00c2ebb8fb424f867424645e944305"
format_stamp: "Formatted at 2026-08-12 07:06:36 on dist-test-slave-w4v5"
server_key: "8b06be26829e9fee21f15ecc08120115"
server_key_iv: "eafe76e1f7e4102b261fcaea5d174e39"
server_key_version: "encryptionkey@0"
I20260812 07:06:36.498955 976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "1c00c2ebb8fb424f867424645e944305"
format_stamp: "Formatted at 2026-08-12 07:06:36 on dist-test-slave-w4v5"
server_key: "8b06be26829e9fee21f15ecc08120115"
server_key_iv: "eafe76e1f7e4102b261fcaea5d174e39"
server_key_version: "encryptionkey@0"
I20260812 07:06:36.502665 976 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260812 07:06:36.505015 1012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:36.506208 976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:06:36.506361 976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "1c00c2ebb8fb424f867424645e944305"
format_stamp: "Formatted at 2026-08-12 07:06:36 on dist-test-slave-w4v5"
server_key: "8b06be26829e9fee21f15ecc08120115"
server_key_iv: "eafe76e1f7e4102b261fcaea5d174e39"
server_key_version: "encryptionkey@0"
I20260812 07:06:36.506470 976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:36.524224 976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:36.527585 976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:36.527801 976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:36.535161 976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37935
I20260812 07:06:36.535187 1129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37935 every 8 connection(s)
I20260812 07:06:36.536216 976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:36.536852 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 976
I20260812 07:06:36.536990 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:36.537317 28091 external_mini_cluster.cc:1428] Setting key a12c940ca8b4b5c40bdb74e622382b3f
I20260812 07:06:36.539569 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:36.545158 1130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305: Bootstrap starting.
Aug 12 07:06:36 dist-test-slave-w4v5 krb5kdc[949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:06:36.548147 1130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:36.548914 1130 log.cc:824] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:36.551281 1130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305: No bootstrap required, opened a new log
I20260812 07:06:36.553218 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:36.538820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:45323 (local address 127.0.0.1:37935)
0812 07:06:36.539306 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:36.539321 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:36.539357 (+ 36us) server_negotiation.cc:407] Connection header received
0812 07:06:36.540087 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:36.540106 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:36.540448 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:36.540783 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:36.541686 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:36.542988 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:36.543978 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:36.544461 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:36.544710 (+ 249us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:36.547771 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:36.547776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:36.547791 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:36.547821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:36.549941 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:36.550590 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:36.550595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:36.550600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:36.550724 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:36.551182 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:36.551185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:36.551186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:36.551658 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:36.551823 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:36.552159 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 07:06:36.552411 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":165,"threads_started":1}
I20260812 07:06:36.555552 1130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c00c2ebb8fb424f867424645e944305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260812 07:06:36.555820 1130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:36.555913 1130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c00c2ebb8fb424f867424645e944305, State: Initialized, Role: FOLLOWER
I20260812 07:06:36.556385 1130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c00c2ebb8fb424f867424645e944305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260812 07:06:36.556504 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:36.556563 1130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:36.556627 1130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:36.556726 1130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:36.557987 1130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c00c2ebb8fb424f867424645e944305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260812 07:06:36.558364 1130 leader_election.cc:302] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c00c2ebb8fb424f867424645e944305; no voters:
I20260812 07:06:36.558739 1130 leader_election.cc:288] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:36.558848 1143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:36.559206 1143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [term 1 LEADER]: Becoming Leader. State: Replica: 1c00c2ebb8fb424f867424645e944305, State: Running, Role: LEADER
I20260812 07:06:36.559546 1143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c00c2ebb8fb424f867424645e944305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } }
I20260812 07:06:36.560175 1130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:36.561200 1152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c00c2ebb8fb424f867424645e944305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c00c2ebb8fb424f867424645e944305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } } }
I20260812 07:06:36.561326 1152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:36.561568 1152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c00c2ebb8fb424f867424645e944305. Latest consensus state: current_term: 1 leader_uuid: "1c00c2ebb8fb424f867424645e944305" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c00c2ebb8fb424f867424645e944305" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37935 } } }
I20260812 07:06:36.561645 1152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:36.561914 1155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:36.565166 1155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:36.572240 1155 catalog_manager.cc:1347] Generated new cluster ID: fe2a7e58cf8e4f10aa18133e74574207
I20260812 07:06:36.572305 1155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:36.572624 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:36.558906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:37589 (local address 127.0.0.1:37935)
0812 07:06:36.559044 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:36.559049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:36.563768 (+ 4719us) server_negotiation.cc:407] Connection header received
0812 07:06:36.564093 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:36.564097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:36.564152 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:36.564256 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:36.565101 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:36.565843 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:36.566700 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:36.566947 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:36.567058 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:36.568896 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:36.568900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:36.568903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:36.568934 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:36.570617 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:36.571137 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:36.571140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:36.571143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:36.571239 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:36.571588 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:36.571591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:36.571593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:36.571781 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:36.571941 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:36.572225 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 07:06:36.572327 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:06:36.581043 1155 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:36.581907 1155 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:36.591425 1155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c00c2ebb8fb424f867424645e944305: Generated new TSK 0
I20260812 07:06:36.592136 1155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:36.600334 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 976
W20260812 07:06:36.619688 946 connection.cc:535] client connection to 127.0.0.1:37935 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37935): Connection reset by peer (error 104)
2026-08-12T07:06:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1695 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:36 dist-test-slave-w4v5 krb5kdc[1196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:36 dist-test-slave-w4v5 krb5kdc[1196](info): set up 2 sockets
Aug 12 07:06:36 dist-test-slave-w4v5 krb5kdc[1196](info): commencing operation
krb5kdc: starting...
W20260812 07:06:37.905290 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:37 dist-test-slave-w4v5 krb5kdc[1196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:38.034157 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35997
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:38.147274 1228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:38.147562 1228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:38.147637 1228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:38.151646 1228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:38.151726 1228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:38.151762 1228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:38.151854 1228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:38.151914 1228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:38.157038 1228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:38.158018 1228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:38.158991 1228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:38.167308 1242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:38.167296 1238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:38.167512 1228 server_base.cc:1034] running on GCE node
W20260812 07:06:38.167308 1237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:38.168166 1228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:38.169202 1228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:38.170363 1228 hybrid_clock.cc:648] HybridClock initialized: now 1786518398170345 us; error 31 us; skew 500 ppm
Aug 12 07:06:38 dist-test-slave-w4v5 krb5kdc[1196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:38.173192 1228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:38.173975 1228 webserver.cc:460] Webserver started at http://127.0.0.1:46237/ using document root <none> and password file <none>
I20260812 07:06:38.174467 1228 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:38.174522 1228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:38.174697 1228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:38.176671 1228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "ea36969ce73348bb8be35ffe9eefba77"
format_stamp: "Formatted at 2026-08-12 07:06:38 on dist-test-slave-w4v5"
server_key: "23db67f0d7b156e908c5c276715e00ce"
server_key_iv: "9986e8e3a8375fceaf2bc711720b021c"
server_key_version: "encryptionkey@0"
I20260812 07:06:38.177160 1228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "ea36969ce73348bb8be35ffe9eefba77"
format_stamp: "Formatted at 2026-08-12 07:06:38 on dist-test-slave-w4v5"
server_key: "23db67f0d7b156e908c5c276715e00ce"
server_key_iv: "9986e8e3a8375fceaf2bc711720b021c"
server_key_version: "encryptionkey@0"
I20260812 07:06:38.180754 1228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:06:38.183020 1251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:38.184265 1228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:38.184413 1228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "ea36969ce73348bb8be35ffe9eefba77"
format_stamp: "Formatted at 2026-08-12 07:06:38 on dist-test-slave-w4v5"
server_key: "23db67f0d7b156e908c5c276715e00ce"
server_key_iv: "9986e8e3a8375fceaf2bc711720b021c"
server_key_version: "encryptionkey@0"
I20260812 07:06:38.184523 1228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:38.218612 1228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:38.221835 1228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:38.222043 1228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:38.229475 1228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35997
I20260812 07:06:38.229511 1360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35997 every 8 connection(s)
I20260812 07:06:38.230422 1228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:38.231213 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1228
I20260812 07:06:38.231350 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:38.231683 28091 external_mini_cluster.cc:1428] Setting key 09f14ddafd9b7cc322efe85c5b742ae4
I20260812 07:06:38.233815 1363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:38.239529 1363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77: Bootstrap starting.
I20260812 07:06:38.240926 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:38.233135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:38365 (local address 127.0.0.1:35997)
0812 07:06:38.233643 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:38.233657 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:38.233693 (+ 36us) server_negotiation.cc:407] Connection header received
0812 07:06:38.234396 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:38.234414 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:38.234757 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:38.235128 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:38.235549 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:38.236538 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:38.237438 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:38.237918 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:38.238095 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:38.239164 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:38.239170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:38.239184 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:38.239612 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:38.239694 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:38.239721 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:38.239864 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:38.240054 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 07:06:38.240198 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":156,"threads_started":1}
I20260812 07:06:38.241943 1363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:38.242722 1363 log.cc:824] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:38.243939 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:38.245011 1363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77: No bootstrap required, opened a new log
I20260812 07:06:38.249792 1363 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea36969ce73348bb8be35ffe9eefba77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 07:06:38.250097 1363 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:38.250185 1363 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea36969ce73348bb8be35ffe9eefba77, State: Initialized, Role: FOLLOWER
I20260812 07:06:38.250685 1368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:38.246308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:32819 (local address 127.0.0.1:35997)
0812 07:06:38.246440 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:38.246444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:38.246679 (+ 235us) server_negotiation.cc:407] Connection header received
0812 07:06:38.246776 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:38.246778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:38.246819 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:38.246886 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:38.247213 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:38.247745 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:38.248913 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:38.249120 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:38.249194 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:38.250059 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:38.250062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:38.250063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:38.250097 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:38.250132 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:38.250148 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:38.250285 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:38.250442 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 07:06:38.250493 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 07:06:38.250681 1363 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea36969ce73348bb8be35ffe9eefba77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 07:06:38.250841 1363 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:38.250905 1363 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:38.251004 1363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:38.252097 1363 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea36969ce73348bb8be35ffe9eefba77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 07:06:38.252398 1363 leader_election.cc:302] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea36969ce73348bb8be35ffe9eefba77; no voters:
I20260812 07:06:38.252646 1363 leader_election.cc:288] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:38.252808 1379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:38.253059 1379 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [term 1 LEADER]: Becoming Leader. State: Replica: ea36969ce73348bb8be35ffe9eefba77, State: Running, Role: LEADER
I20260812 07:06:38.253381 1379 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea36969ce73348bb8be35ffe9eefba77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } }
I20260812 07:06:38.253857 1363 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:38.255148 1384 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea36969ce73348bb8be35ffe9eefba77. Latest consensus state: current_term: 1 leader_uuid: "ea36969ce73348bb8be35ffe9eefba77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea36969ce73348bb8be35ffe9eefba77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } } }
I20260812 07:06:38.255162 1381 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea36969ce73348bb8be35ffe9eefba77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea36969ce73348bb8be35ffe9eefba77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35997 } } }
I20260812 07:06:38.255295 1381 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:38.255295 1384 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:38.255728 1399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:38.259034 1399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:38.264271 1399 catalog_manager.cc:1347] Generated new cluster ID: c51f08ed9bdc4eaa8792647713ae5f0c
I20260812 07:06:38.264338 1399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:38.284458 1399 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:38.285441 1399 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:38.308126 1399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea36969ce73348bb8be35ffe9eefba77: Generated new TSK 0
I20260812 07:06:38.308871 1399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:38.317049 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1228
W20260812 07:06:38.335340 1192 connection.cc:535] client connection to 127.0.0.1:35997 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35997): Connection reset by peer (error 104)
2026-08-12T07:06:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1715 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:38 dist-test-slave-w4v5 krb5kdc[1426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:38 dist-test-slave-w4v5 krb5kdc[1426](info): set up 2 sockets
Aug 12 07:06:38 dist-test-slave-w4v5 krb5kdc[1426](info): commencing operation
krb5kdc: starting...
W20260812 07:06:39.649498 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.299s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:39 dist-test-slave-w4v5 krb5kdc[1426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:39.790400 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:39.907958 1455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:39.908264 1455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:39.908350 1455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:39.912417 1455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:39.912520 1455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:39.912566 1455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:39.912634 1455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:39.912671 1455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:39.918069 1455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:39.919241 1455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:39.920296 1455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:39.929116 1468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:39.929137 1467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:39.929267 1476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:39.929293 1455 server_base.cc:1034] running on GCE node
I20260812 07:06:39.930075 1455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:39.931351 1455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:39.932525 1455 hybrid_clock.cc:648] HybridClock initialized: now 1786518399932504 us; error 34 us; skew 500 ppm
Aug 12 07:06:39 dist-test-slave-w4v5 krb5kdc[1426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:39.935626 1455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:39.936621 1455 webserver.cc:460] Webserver started at http://127.0.0.1:43695/ using document root <none> and password file <none>
I20260812 07:06:39.937202 1455 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:39.937266 1455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:39.937510 1455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:39.939672 1455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "5b709d69a219465494191538200f53e7"
format_stamp: "Formatted at 2026-08-12 07:06:39 on dist-test-slave-w4v5"
server_key: "381557a2fcde6373b97d13e67e19aa7e"
server_key_iv: "44b77bed738efe2e0b3f9edfb1b354c1"
server_key_version: "encryptionkey@0"
I20260812 07:06:39.940236 1455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "5b709d69a219465494191538200f53e7"
format_stamp: "Formatted at 2026-08-12 07:06:39 on dist-test-slave-w4v5"
server_key: "381557a2fcde6373b97d13e67e19aa7e"
server_key_iv: "44b77bed738efe2e0b3f9edfb1b354c1"
server_key_version: "encryptionkey@0"
I20260812 07:06:39.944154 1455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:06:39.946578 1486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:39.947888 1455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:39.948038 1455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "5b709d69a219465494191538200f53e7"
format_stamp: "Formatted at 2026-08-12 07:06:39 on dist-test-slave-w4v5"
server_key: "381557a2fcde6373b97d13e67e19aa7e"
server_key_iv: "44b77bed738efe2e0b3f9edfb1b354c1"
server_key_version: "encryptionkey@0"
I20260812 07:06:39.948168 1455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:39.952986 1455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:39.956460 1455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:39.956679 1455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:39.964465 1455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43977
I20260812 07:06:39.964493 1602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43977 every 8 connection(s)
I20260812 07:06:39.965431 1455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:39.968070 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1455
I20260812 07:06:39.968238 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:39.968379 1605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:39.968600 28091 external_mini_cluster.cc:1428] Setting key 123f7d88d6f44959935739cc54338054
I20260812 07:06:39.974283 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:39.970042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:50307 (local address 127.0.0.1:43977)
0812 07:06:39.970506 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:39.970526 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:39.970570 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:06:39.971305 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:39.971328 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:39.971702 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:39.972082 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:39.972872 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:39.972878 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:39.972893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:39.972905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:39.972995 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:39.973023 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:39.973164 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:39.973230 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 07:06:39.973394 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":133,"threads_started":1}
I20260812 07:06:39.976082 1605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7: Bootstrap starting.
I20260812 07:06:39.977195 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:39.978626 1605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:39.979490 1605 log.cc:824] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:39.981259 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:39.979377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:33521 (local address 127.0.0.1:43977)
0812 07:06:39.979509 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:39.979513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:39.979801 (+ 288us) server_negotiation.cc:407] Connection header received
0812 07:06:39.979874 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:39.979878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:39.979912 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:39.979992 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:39.980789 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:39.980792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:39.980795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:39.980806 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:39.980834 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:39.980846 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:39.980938 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:39.981103 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 07:06:39.981152 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 07:06:39.981752 1605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7: No bootstrap required, opened a new log
I20260812 07:06:39.985281 1605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b709d69a219465494191538200f53e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260812 07:06:39.985522 1605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:39.985603 1605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b709d69a219465494191538200f53e7, State: Initialized, Role: FOLLOWER
I20260812 07:06:39.986029 1605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b709d69a219465494191538200f53e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260812 07:06:39.986191 1605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:39.986279 1605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:39.986411 1605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:39.987381 1605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b709d69a219465494191538200f53e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260812 07:06:39.987739 1605 leader_election.cc:302] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b709d69a219465494191538200f53e7; no voters:
I20260812 07:06:39.988121 1605 leader_election.cc:288] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:39.988277 1630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:39.988569 1630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [term 1 LEADER]: Becoming Leader. State: Replica: 5b709d69a219465494191538200f53e7, State: Running, Role: LEADER
I20260812 07:06:39.988895 1630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b709d69a219465494191538200f53e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } }
I20260812 07:06:39.989435 1605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:39.990803 1631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b709d69a219465494191538200f53e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b709d69a219465494191538200f53e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } } }
I20260812 07:06:39.990806 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b709d69a219465494191538200f53e7. Latest consensus state: current_term: 1 leader_uuid: "5b709d69a219465494191538200f53e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b709d69a219465494191538200f53e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43977 } } }
I20260812 07:06:39.990944 1631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:39.990955 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:39.991367 1652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:39.994230 1652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:39.999370 1652 catalog_manager.cc:1347] Generated new cluster ID: 21ab124ef9ce4210835c3ef47b8b9e75
I20260812 07:06:39.999435 1652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:40.011793 1652 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:40.012116 1652 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:40.017156 1652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b709d69a219465494191538200f53e7: Generated new TSK 0
I20260812 07:06:40.017860 1652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:40.026684 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1455
2026-08-12T07:06:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1705 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:40 dist-test-slave-w4v5 krb5kdc[1682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:40 dist-test-slave-w4v5 krb5kdc[1682](info): set up 2 sockets
Aug 12 07:06:40 dist-test-slave-w4v5 krb5kdc[1682](info): commencing operation
krb5kdc: starting...
W20260812 07:06:41.351130 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.296s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:41 dist-test-slave-w4v5 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:41.487044 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40033
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:41.603204 1711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:41.603487 1711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:41.603572 1711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:41.607936 1711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:41.608006 1711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:41.608027 1711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:41.608120 1711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:41.608157 1711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:41.613742 1711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40033
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:41.614864 1711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:41.615988 1711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:41.624610 1716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:41.624621 1723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:41.624732 1720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:41.625061 1711 server_base.cc:1034] running on GCE node
I20260812 07:06:41.625671 1711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:41.626883 1711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:41.628089 1711 hybrid_clock.cc:648] HybridClock initialized: now 1786518401628074 us; error 35 us; skew 500 ppm
Aug 12 07:06:41 dist-test-slave-w4v5 krb5kdc[1682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:41.637769 1711 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:41.638763 1711 webserver.cc:460] Webserver started at http://127.0.0.1:42531/ using document root <none> and password file <none>
I20260812 07:06:41.639381 1711 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:41.639448 1711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:41.639624 1711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:41.641696 1711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "78018520d1a949cd8fba73cdfacf2fa8"
format_stamp: "Formatted at 2026-08-12 07:06:41 on dist-test-slave-w4v5"
server_key: "81437151fb9e38416b9bb6dac50f8a61"
server_key_iv: "9ef1529917be47109e995ed72af4360c"
server_key_version: "encryptionkey@0"
I20260812 07:06:41.642215 1711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "78018520d1a949cd8fba73cdfacf2fa8"
format_stamp: "Formatted at 2026-08-12 07:06:41 on dist-test-slave-w4v5"
server_key: "81437151fb9e38416b9bb6dac50f8a61"
server_key_iv: "9ef1529917be47109e995ed72af4360c"
server_key_version: "encryptionkey@0"
I20260812 07:06:41.645936 1711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:06:41.648350 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:41.649631 1711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:41.649763 1711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "78018520d1a949cd8fba73cdfacf2fa8"
format_stamp: "Formatted at 2026-08-12 07:06:41 on dist-test-slave-w4v5"
server_key: "81437151fb9e38416b9bb6dac50f8a61"
server_key_iv: "9ef1529917be47109e995ed72af4360c"
server_key_version: "encryptionkey@0"
I20260812 07:06:41.649870 1711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:41.668145 1711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:41.671330 1711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:41.671538 1711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:41.679255 1711 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40033
I20260812 07:06:41.679275 1838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40033 every 8 connection(s)
I20260812 07:06:41.680330 1711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:41.683187 1842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:41.684616 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1711
I20260812 07:06:41.684763 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:41.685115 28091 external_mini_cluster.cc:1428] Setting key ab695b7bd1b4126b41b19cf0ef25a04b
I20260812 07:06:41.689513 1842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8: Bootstrap starting.
I20260812 07:06:41.691817 1842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:41.692649 1842 log.cc:824] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:06:41 dist-test-slave-w4v5 krb5kdc[1682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 07:06:41.694774 1842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8: No bootstrap required, opened a new log
I20260812 07:06:41.698652 1842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78018520d1a949cd8fba73cdfacf2fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40033 } }
I20260812 07:06:41.698908 1842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:41.698969 1842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78018520d1a949cd8fba73cdfacf2fa8, State: Initialized, Role: FOLLOWER
I20260812 07:06:41.699487 1842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78018520d1a949cd8fba73cdfacf2fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40033 } }
I20260812 07:06:41.699651 1842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:41.699739 1842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:41.699841 1842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:41.700027 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:41.686608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:46569 (local address 127.0.0.1:40033)
0812 07:06:41.687160 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:41.687179 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:41.687217 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:06:41.687868 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:41.687885 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:41.688239 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:41.688572 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:41.689396 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:41.690421 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:41.691405 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:41.691907 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:41.692088 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:41.694930 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:41.694936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:41.694950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:41.694978 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:41.696850 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:41.697378 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:41.697384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:41.697389 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:41.697502 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:41.697865 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:41.697873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:41.697874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:41.698267 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:41.698518 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:41.698802 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 07:06:41.699032 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":136,"threads_started":1}
I20260812 07:06:41.700830 1842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78018520d1a949cd8fba73cdfacf2fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40033 } }
I20260812 07:06:41.701205 1842 leader_election.cc:302] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78018520d1a949cd8fba73cdfacf2fa8; no voters:
I20260812 07:06:41.701558 1842 leader_election.cc:288] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:41.701735 1851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:41.702010 1851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [term 1 LEADER]: Becoming Leader. State: Replica: 78018520d1a949cd8fba73cdfacf2fa8, State: Running, Role: LEADER
I20260812 07:06:41.702348 1851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78018520d1a949cd8fba73cdfacf2fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40033 } }
I20260812 07:06:41.702939 1842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:41.703253 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:41.704113 1855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78018520d1a949cd8fba73cdfacf2fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78018520d1a949cd8fba73cdfacf2fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40033 } } }
I20260812 07:06:41.704242 1855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:41.704603 1858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78018520d1a949cd8fba73cdfacf2fa8. Latest consensus state: current_term: 1 leader_uuid: "78018520d1a949cd8fba73cdfacf2fa8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78018520d1a949cd8fba73cdfacf2fa8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40033 } } }
I20260812 07:06:41.704694 1858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8 [sys.catalog]: This master's current role is: LEADER
W20260812 07:06:41.708380 1902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:41.708523 1902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:41.708631 1879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:41.709527 1879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:41.715212 1847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:41.706167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:51117 (local address 127.0.0.1:40033)
0812 07:06:41.706283 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:41.706287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:41.706475 (+ 188us) server_negotiation.cc:407] Connection header received
0812 07:06:41.706582 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:41.706584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:41.706623 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:41.706688 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:41.707587 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:41.708360 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:41.709308 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:41.709554 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:41.709674 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:41.711272 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:41.711276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:41.711279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:41.711310 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:41.713108 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:41.713686 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:41.713693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:41.713695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:41.713786 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:41.714186 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:41.714190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:41.714192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:41.714434 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:41.714607 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:41.714889 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 07:06:41.714991 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:06:41.715952 1879 catalog_manager.cc:1347] Generated new cluster ID: 12b5067e409b4a3cace5c2b51fce7d4b
I20260812 07:06:41.716014 1879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:41.731781 1879 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:41.733167 1879 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:41.754401 1879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78018520d1a949cd8fba73cdfacf2fa8: Generated new TSK 0
I20260812 07:06:41.755370 1879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:41.764885 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1711
W20260812 07:06:41.780005 1678 connection.cc:535] client connection to 127.0.0.1:40033 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40033): Connection reset by peer (error 104)
2026-08-12T07:06:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1739 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:41 dist-test-slave-w4v5 krb5kdc[1916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:41 dist-test-slave-w4v5 krb5kdc[1916](info): set up 2 sockets
Aug 12 07:06:41 dist-test-slave-w4v5 krb5kdc[1916](info): commencing operation
krb5kdc: starting...
W20260812 07:06:43.079797 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:43 dist-test-slave-w4v5 krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:43.214476 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:43.331029 1939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:43.331305 1939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:43.331347 1939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:43.335330 1939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:43.335388 1939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:43.335409 1939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:43.335425 1939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:43.335440 1939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:43.340637 1939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:43.341676 1939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:43.342685 1939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:43.351447 1950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:43.351449 1944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:43.351447 1948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:43.351589 1939 server_base.cc:1034] running on GCE node
I20260812 07:06:43.352398 1939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:43.353551 1939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:43.354766 1939 hybrid_clock.cc:648] HybridClock initialized: now 1786518403354751 us; error 27 us; skew 500 ppm
Aug 12 07:06:43 dist-test-slave-w4v5 krb5kdc[1916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:43.357717 1939 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:43.358626 1939 webserver.cc:460] Webserver started at http://127.0.0.1:44087/ using document root <none> and password file <none>
I20260812 07:06:43.359283 1939 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:43.359387 1939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:43.359632 1939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:43.361814 1939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "5bee699feacd4dfbb7446bc6952c33a9"
format_stamp: "Formatted at 2026-08-12 07:06:43 on dist-test-slave-w4v5"
server_key: "c3e45e4f8540fda17bc3da8b9aad2566"
server_key_iv: "0bfb8a1ee461789205bdb1270fd9c11b"
server_key_version: "encryptionkey@0"
I20260812 07:06:43.362399 1939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "5bee699feacd4dfbb7446bc6952c33a9"
format_stamp: "Formatted at 2026-08-12 07:06:43 on dist-test-slave-w4v5"
server_key: "c3e45e4f8540fda17bc3da8b9aad2566"
server_key_iv: "0bfb8a1ee461789205bdb1270fd9c11b"
server_key_version: "encryptionkey@0"
I20260812 07:06:43.366271 1939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260812 07:06:43.368747 1959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:43.369977 1939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260812 07:06:43.370134 1939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "5bee699feacd4dfbb7446bc6952c33a9"
format_stamp: "Formatted at 2026-08-12 07:06:43 on dist-test-slave-w4v5"
server_key: "c3e45e4f8540fda17bc3da8b9aad2566"
server_key_iv: "0bfb8a1ee461789205bdb1270fd9c11b"
server_key_version: "encryptionkey@0"
I20260812 07:06:43.370247 1939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:43.385706 1939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:43.388598 1939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:43.388821 1939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:43.396726 1939 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37401
I20260812 07:06:43.396751 2067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37401 every 8 connection(s)
I20260812 07:06:43.397697 1939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:43.400610 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:43.402143 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 1939
I20260812 07:06:43.402280 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:43.402599 28091 external_mini_cluster.cc:1428] Setting key e9ce7465af6ad78b51e9f0a1b0870f4c
I20260812 07:06:43.406842 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9: Bootstrap starting.
I20260812 07:06:43.409070 2069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:43.409940 2069 log.cc:824] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:43.411733 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:43.404077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:47245 (local address 127.0.0.1:37401)
0812 07:06:43.404575 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:43.404593 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:43.404631 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:06:43.405270 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:43.405287 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:43.405626 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:43.405979 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:43.406387 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:43.407470 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:43.408388 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:43.408855 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:43.409025 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:43.409945 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:43.409950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:43.409963 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:43.410394 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:43.410462 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:43.410486 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:43.410655 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:43.410840 (+ 185us) server_negotiation.cc:299] Negotiation successful
0812 07:06:43.410979 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":137,"threads_started":1}
I20260812 07:06:43.412024 2069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9: No bootstrap required, opened a new log
I20260812 07:06:43.414639 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:43.415756 2069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bee699feacd4dfbb7446bc6952c33a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260812 07:06:43.416038 2069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:43.416105 2069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bee699feacd4dfbb7446bc6952c33a9, State: Initialized, Role: FOLLOWER
I20260812 07:06:43.416563 2069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bee699feacd4dfbb7446bc6952c33a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260812 07:06:43.416716 2069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:43.416782 2069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:43.416884 2069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:43.417786 2069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bee699feacd4dfbb7446bc6952c33a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260812 07:06:43.418115 2069 leader_election.cc:302] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bee699feacd4dfbb7446bc6952c33a9; no voters:
I20260812 07:06:43.418407 2069 leader_election.cc:288] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:43.418525 2086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:43.418848 2086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [term 1 LEADER]: Becoming Leader. State: Replica: 5bee699feacd4dfbb7446bc6952c33a9, State: Running, Role: LEADER
I20260812 07:06:43.419220 2086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bee699feacd4dfbb7446bc6952c33a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } }
I20260812 07:06:43.419828 2069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:43.420990 2087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bee699feacd4dfbb7446bc6952c33a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bee699feacd4dfbb7446bc6952c33a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } } }
I20260812 07:06:43.421125 2087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:43.421512 2090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bee699feacd4dfbb7446bc6952c33a9. Latest consensus state: current_term: 1 leader_uuid: "5bee699feacd4dfbb7446bc6952c33a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bee699feacd4dfbb7446bc6952c33a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37401 } } }
I20260812 07:06:43.421658 2090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:43.424129 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:43.416758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:49381 (local address 127.0.0.1:37401)
0812 07:06:43.416893 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:43.416897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:43.419356 (+ 2459us) server_negotiation.cc:407] Connection header received
0812 07:06:43.419545 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:43.419548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:43.419598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:43.419697 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:43.420083 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:43.420943 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:43.422325 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:43.422649 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:43.422740 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:43.423629 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:43.423632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:43.423635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:43.423648 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:43.423675 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:43.423688 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:43.423782 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:43.423954 (+ 172us) server_negotiation.cc:299] Negotiation successful
0812 07:06:43.423993 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
W20260812 07:06:43.425329 2133 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:43.425410 2133 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:43.425510 2117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:43.426214 2117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:43.431406 2117 catalog_manager.cc:1347] Generated new cluster ID: a967c215f1a844d183f5dc8151828de7
I20260812 07:06:43.431470 2117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:43.460067 2117 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:43.461057 2117 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:43.469064 2117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bee699feacd4dfbb7446bc6952c33a9: Generated new TSK 0
I20260812 07:06:43.469791 2117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:43.474097 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 1939
W20260812 07:06:43.493516 1913 connection.cc:535] client connection to 127.0.0.1:37401 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37401): Connection reset by peer (error 104)
2026-08-12T07:06:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1713 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:43 dist-test-slave-w4v5 krb5kdc[2145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:43 dist-test-slave-w4v5 krb5kdc[2145](info): set up 2 sockets
Aug 12 07:06:43 dist-test-slave-w4v5 krb5kdc[2145](info): commencing operation
krb5kdc: starting...
W20260812 07:06:44.794368 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:44 dist-test-slave-w4v5 krb5kdc[2145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 07:06:44.930361 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:45.046497 2170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:45.046837 2170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:45.046916 2170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:45.050961 2170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:45.051074 2170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:45.051124 2170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:45.051167 2170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:45.051204 2170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:45.056435 2170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:45.057551 2170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:45.058586 2170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:45.067411 2178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:45.067433 2181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:45.067420 2179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:45.067569 2170 server_base.cc:1034] running on GCE node
I20260812 07:06:45.068331 2170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:45.069514 2170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:45.070716 2170 hybrid_clock.cc:648] HybridClock initialized: now 1786518405070682 us; error 51 us; skew 500 ppm
Aug 12 07:06:45 dist-test-slave-w4v5 krb5kdc[2145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:45.073800 2170 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 07:06:45.074800 2170 webserver.cc:460] Webserver started at http://127.0.0.1:46611/ using document root <none> and password file <none>
I20260812 07:06:45.075444 2170 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:45.075541 2170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:45.075793 2170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:45.077983 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88"
format_stamp: "Formatted at 2026-08-12 07:06:45 on dist-test-slave-w4v5"
server_key: "8acfc693f670452d44537e6f81f7cd96"
server_key_iv: "4cd1bea37c1e6e3a7261076142c6efa6"
server_key_version: "encryptionkey@0"
I20260812 07:06:45.078563 2170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88"
format_stamp: "Formatted at 2026-08-12 07:06:45 on dist-test-slave-w4v5"
server_key: "8acfc693f670452d44537e6f81f7cd96"
server_key_iv: "4cd1bea37c1e6e3a7261076142c6efa6"
server_key_version: "encryptionkey@0"
I20260812 07:06:45.082397 2170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260812 07:06:45.085019 2193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:45.086292 2170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:45.086465 2170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88"
format_stamp: "Formatted at 2026-08-12 07:06:45 on dist-test-slave-w4v5"
server_key: "8acfc693f670452d44537e6f81f7cd96"
server_key_iv: "4cd1bea37c1e6e3a7261076142c6efa6"
server_key_version: "encryptionkey@0"
I20260812 07:06:45.086586 2170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:45.091624 2170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:45.094707 2170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:45.094910 2170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:45.102635 2170 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34121
I20260812 07:06:45.102661 2297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34121 every 8 connection(s)
I20260812 07:06:45.103713 2170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:45.106669 2299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:45.108161 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2170
I20260812 07:06:45.108315 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:45.108618 28091 external_mini_cluster.cc:1428] Setting key a0e5ecb9dc5a6f076e795445abdde7bc
I20260812 07:06:45.114056 2299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88: Bootstrap starting.
I20260812 07:06:45.114743 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:45.110159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:51491 (local address 127.0.0.1:34121)
0812 07:06:45.110646 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:45.110666 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:45.110711 (+ 45us) server_negotiation.cc:407] Connection header received
0812 07:06:45.111579 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:45.111603 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:45.111982 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:45.112372 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:45.113219 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:45.113225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:45.113240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:45.113252 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:45.113319 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:45.113346 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:45.113494 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:45.113573 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 07:06:45.113730 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":130,"threads_started":1}
I20260812 07:06:45.116815 2299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:45.117751 2299 log.cc:824] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:45.118000 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:45.120131 2299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88: No bootstrap required, opened a new log
I20260812 07:06:45.122337 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:45.120555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.75:48017 (local address 127.0.0.1:34121)
0812 07:06:45.120705 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:45.120710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:45.120905 (+ 195us) server_negotiation.cc:407] Connection header received
0812 07:06:45.121035 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:45.121038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:45.121070 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:45.121133 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:45.121924 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:45.121927 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:45.121929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:45.121940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:45.121965 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:45.121975 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:45.122071 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:45.122157 (+ 86us) server_negotiation.cc:299] Negotiation successful
0812 07:06:45.122190 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 07:06:45.124722 2299 raft_consensus.cc:348] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260812 07:06:45.124976 2299 raft_consensus.cc:374] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:45.125046 2299 raft_consensus.cc:729] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a441f21d8eb04f4d9bc26dd3dbfb4c88, State: Initialized, Role: FOLLOWER
I20260812 07:06:45.125481 2299 consensus_queue.cc:260] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260812 07:06:45.125639 2299 raft_consensus.cc:388] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:45.125708 2299 raft_consensus.cc:482] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:45.125808 2299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:45.126749 2299 raft_consensus.cc:504] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260812 07:06:45.127161 2299 leader_election.cc:302] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a441f21d8eb04f4d9bc26dd3dbfb4c88; no voters:
I20260812 07:06:45.127523 2299 leader_election.cc:288] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:45.127596 2321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:45.127892 2321 raft_consensus.cc:686] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [term 1 LEADER]: Becoming Leader. State: Replica: a441f21d8eb04f4d9bc26dd3dbfb4c88, State: Running, Role: LEADER
I20260812 07:06:45.128247 2321 consensus_queue.cc:237] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } }
I20260812 07:06:45.128832 2299 sys_catalog.cc:564] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:45.130237 2323 sys_catalog.cc:455] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } } }
I20260812 07:06:45.130242 2325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a441f21d8eb04f4d9bc26dd3dbfb4c88. Latest consensus state: current_term: 1 leader_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a441f21d8eb04f4d9bc26dd3dbfb4c88" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34121 } } }
I20260812 07:06:45.130456 2325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:45.130439 2323 sys_catalog.cc:458] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:45.131445 2337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:45.134219 2337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:45.140100 2337 catalog_manager.cc:1347] Generated new cluster ID: deb5e796e03849bf82bdbc4f7e6d3850
I20260812 07:06:45.140184 2337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:45.166478 2337 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:45.166844 2337 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:45.173401 2337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a441f21d8eb04f4d9bc26dd3dbfb4c88: Generated new TSK 0
I20260812 07:06:45.174238 2337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:45.178211 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2170
2026-08-12T07:06:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1701 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:45 dist-test-slave-w4v5 krb5kdc[2361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:45 dist-test-slave-w4v5 krb5kdc[2361](info): set up 2 sockets
Aug 12 07:06:45 dist-test-slave-w4v5 krb5kdc[2361](info): commencing operation
krb5kdc: starting...
W20260812 07:06:46.501897 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:46 dist-test-slave-w4v5 krb5kdc[2361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:46.637959 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44271
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:40475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:44271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:46.751502 2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:46.751839 2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:46.751915 2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:46.755903 2383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:46.755992 2383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:46.756031 2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:46.756089 2383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:46.756130 2383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:46.764065 2383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:44271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:46.765177 2383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:46.766256 2383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:46.775065 2393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:46.775202 2398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:46.775239 2383 server_base.cc:1034] running on GCE node
W20260812 07:06:46.775202 2396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:46.776037 2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:46.777092 2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:46.778298 2383 hybrid_clock.cc:648] HybridClock initialized: now 1786518406778259 us; error 49 us; skew 500 ppm
Aug 12 07:06:46 dist-test-slave-w4v5 krb5kdc[2361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518406, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:46.784565 2383 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:46.785559 2383 webserver.cc:460] Webserver started at http://127.27.110.254:40511/ using document root <none> and password file <none>
I20260812 07:06:46.786091 2383 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:46.786163 2383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:46.786350 2383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:46.788558 2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "d072a99cd36c40f1ac365610815327a6"
format_stamp: "Formatted at 2026-08-12 07:06:46 on dist-test-slave-w4v5"
server_key: "7bc7115d094c9443b2688976f6f6e940"
server_key_iv: "dbc53feffd1cf00df79307f697efe2a8"
server_key_version: "encryptionkey@0"
I20260812 07:06:46.789095 2383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "d072a99cd36c40f1ac365610815327a6"
format_stamp: "Formatted at 2026-08-12 07:06:46 on dist-test-slave-w4v5"
server_key: "7bc7115d094c9443b2688976f6f6e940"
server_key_iv: "dbc53feffd1cf00df79307f697efe2a8"
server_key_version: "encryptionkey@0"
I20260812 07:06:46.792979 2383 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260812 07:06:46.795609 2411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:46.796839 2383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:46.796984 2383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "d072a99cd36c40f1ac365610815327a6"
format_stamp: "Formatted at 2026-08-12 07:06:46 on dist-test-slave-w4v5"
server_key: "7bc7115d094c9443b2688976f6f6e940"
server_key_iv: "dbc53feffd1cf00df79307f697efe2a8"
server_key_version: "encryptionkey@0"
I20260812 07:06:46.797097 2383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:46.811442 2383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:46.814436 2383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:46.814661 2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:46.822121 2383 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:44271
I20260812 07:06:46.822137 2516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:44271 every 8 connection(s)
I20260812 07:06:46.823174 2383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:46.825615 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2383
I20260812 07:06:46.825780 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:46.826071 28091 external_mini_cluster.cc:1428] Setting key 51ed3b772366be699842a35cdcdcc36a
I20260812 07:06:46.826133 2520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:46.833710 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6: Bootstrap starting.
Aug 12 07:06:46 dist-test-slave-w4v5 krb5kdc[2361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:46.836242 2520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:46.837196 2520 log.cc:824] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:46.839736 2520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6: No bootstrap required, opened a new log
I20260812 07:06:46.842531 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:46.827634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57621 (local address 127.27.110.254:44271)
0812 07:06:46.828087 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:46.828107 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:46.828151 (+ 44us) server_negotiation.cc:407] Connection header received
0812 07:06:46.829027 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:46.829052 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:46.829496 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:46.829888 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:46.830659 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:46.832177 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:46.833083 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:46.833778 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:46.833951 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:46.836673 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:46.836679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:46.836695 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:46.836728 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:46.839002 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:46.839596 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:46.839602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:46.839608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:46.839767 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:46.840188 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:46.840192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:46.840194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:46.840714 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:46.840928 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:46.841226 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 07:06:46.841461 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":119,"threads_started":1}
I20260812 07:06:46.844412 2520 raft_consensus.cc:348] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d072a99cd36c40f1ac365610815327a6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44271 } }
I20260812 07:06:46.844700 2520 raft_consensus.cc:374] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:46.844777 2520 raft_consensus.cc:729] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d072a99cd36c40f1ac365610815327a6, State: Initialized, Role: FOLLOWER
I20260812 07:06:46.845263 2520 consensus_queue.cc:260] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d072a99cd36c40f1ac365610815327a6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44271 } }
I20260812 07:06:46.845412 2520 raft_consensus.cc:388] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:46.845489 2520 raft_consensus.cc:482] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:46.845607 2520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:46.845816 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:46.846606 2520 raft_consensus.cc:504] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d072a99cd36c40f1ac365610815327a6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44271 } }
I20260812 07:06:46.846971 2520 leader_election.cc:302] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d072a99cd36c40f1ac365610815327a6; no voters:
I20260812 07:06:46.847360 2520 leader_election.cc:288] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:46.847434 2539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:46.847652 2539 raft_consensus.cc:686] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [term 1 LEADER]: Becoming Leader. State: Replica: d072a99cd36c40f1ac365610815327a6, State: Running, Role: LEADER
I20260812 07:06:46.848003 2539 consensus_queue.cc:237] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d072a99cd36c40f1ac365610815327a6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44271 } }
I20260812 07:06:46.848691 2520 sys_catalog.cc:564] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:46.849718 2540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d072a99cd36c40f1ac365610815327a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d072a99cd36c40f1ac365610815327a6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44271 } } }
I20260812 07:06:46.849860 2540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:46.850111 2540 sys_catalog.cc:455] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d072a99cd36c40f1ac365610815327a6. Latest consensus state: current_term: 1 leader_uuid: "d072a99cd36c40f1ac365610815327a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d072a99cd36c40f1ac365610815327a6" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44271 } } }
I20260812 07:06:46.850200 2540 sys_catalog.cc:458] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:46.851450 2562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:46.854431 2562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:46.857858 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:46.848048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57763 (local address 127.27.110.254:44271)
0812 07:06:46.848161 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:46.848165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:46.848623 (+ 458us) server_negotiation.cc:407] Connection header received
0812 07:06:46.848759 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:46.848762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:46.848805 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:46.848897 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:46.849754 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:46.850554 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:46.851894 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:46.852218 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:46.852334 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:46.854058 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:46.854062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:46.854066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:46.854091 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:46.855819 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:46.856390 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:46.856396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:46.856401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:46.856502 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:46.856927 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:46.856930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:46.856932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:46.857147 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:46.857292 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:46.857610 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 07:06:46.857711 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260812 07:06:46.860370 2562 catalog_manager.cc:1347] Generated new cluster ID: cf5e162320c443bc912bd01707188073
I20260812 07:06:46.860483 2562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:46.871943 2562 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:46.872870 2562 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:46.878324 2562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d072a99cd36c40f1ac365610815327a6: Generated new TSK 0
I20260812 07:06:46.879004 2562 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:46.887535 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2383
W20260812 07:06:46.900372 2352 connection.cc:535] client connection to 127.27.110.254:44271 recv error: Network error: failed to read from TLS socket (remote: 127.27.110.254:44271): Connection reset by peer (error 104)
2026-08-12T07:06:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1705 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:46 dist-test-slave-w4v5 krb5kdc[2583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:46 dist-test-slave-w4v5 krb5kdc[2583](info): set up 2 sockets
Aug 12 07:06:46 dist-test-slave-w4v5 krb5kdc[2583](info): commencing operation
krb5kdc: starting...
W20260812 07:06:48.190104 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:48 dist-test-slave-w4v5 krb5kdc[2583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:48.319908 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42211
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:34053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:48.429625 2608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:48.429920 2608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:48.429999 2608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:48.433933 2608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:48.434020 2608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:48.434060 2608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:48.434118 2608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:48.434158 2608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:48.439167 2608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:34053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:48.440182 2608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:48.441167 2608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:48.449831 2615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:48.449820 2616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:48.449924 2608 server_base.cc:1034] running on GCE node
W20260812 07:06:48.450109 2618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:48.450687 2608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:48.451756 2608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:48.452919 2608 hybrid_clock.cc:648] HybridClock initialized: now 1786518408452904 us; error 28 us; skew 500 ppm
Aug 12 07:06:48 dist-test-slave-w4v5 krb5kdc[2583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518408, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:48.455659 2608 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:48.456490 2608 webserver.cc:460] Webserver started at http://127.27.110.254:39165/ using document root <none> and password file <none>
I20260812 07:06:48.457028 2608 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:48.457129 2608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:48.457381 2608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:48.459401 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "b990deab46154acda7b3b3860d648935"
format_stamp: "Formatted at 2026-08-12 07:06:48 on dist-test-slave-w4v5"
server_key: "12f068dff506fbf3073fd87e420d05ea"
server_key_iv: "2950472a3e77bb9244f5d672f68d95d8"
server_key_version: "encryptionkey@0"
I20260812 07:06:48.459960 2608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "b990deab46154acda7b3b3860d648935"
format_stamp: "Formatted at 2026-08-12 07:06:48 on dist-test-slave-w4v5"
server_key: "12f068dff506fbf3073fd87e420d05ea"
server_key_iv: "2950472a3e77bb9244f5d672f68d95d8"
server_key_version: "encryptionkey@0"
I20260812 07:06:48.463624 2608 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:48.466006 2634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:48.467300 2608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:48.467442 2608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "b990deab46154acda7b3b3860d648935"
format_stamp: "Formatted at 2026-08-12 07:06:48 on dist-test-slave-w4v5"
server_key: "12f068dff506fbf3073fd87e420d05ea"
server_key_iv: "2950472a3e77bb9244f5d672f68d95d8"
server_key_version: "encryptionkey@0"
I20260812 07:06:48.467553 2608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:48.487360 2608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:48.490242 2608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:48.490473 2608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:48.497664 2608 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42211
I20260812 07:06:48.497692 2756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42211 every 8 connection(s)
I20260812 07:06:48.498616 2608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:48.501365 2764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:48.506034 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2608
I20260812 07:06:48.506192 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:48.506501 28091 external_mini_cluster.cc:1428] Setting key 38da42f5df2cd1d92d15f25468272fc0
I20260812 07:06:48.507182 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935: Bootstrap starting.
I20260812 07:06:48.509518 2764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:48.510357 2764 log.cc:824] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:48.512648 2764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935: No bootstrap required, opened a new log
I20260812 07:06:48.515853 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:48.507874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45131 (local address 127.27.110.254:42211)
0812 07:06:48.508275 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:48.508291 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:48.508325 (+ 34us) server_negotiation.cc:407] Connection header received
0812 07:06:48.508962 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:48.508980 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:48.509318 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:48.509615 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:48.509970 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:48.511237 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:48.512461 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:48.513098 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:48.513226 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:48.514158 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:48.514162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:48.514174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:48.514548 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:48.514602 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:48.514624 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:48.514790 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:48.514946 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 07:06:48.515135 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":142,"threads_started":1}
I20260812 07:06:48.516644 2764 raft_consensus.cc:348] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b990deab46154acda7b3b3860d648935" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42211 } }
I20260812 07:06:48.516903 2764 raft_consensus.cc:374] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:48.516975 2764 raft_consensus.cc:729] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b990deab46154acda7b3b3860d648935, State: Initialized, Role: FOLLOWER
I20260812 07:06:48.517433 2764 consensus_queue.cc:260] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b990deab46154acda7b3b3860d648935" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42211 } }
I20260812 07:06:48.517594 2764 raft_consensus.cc:388] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:48.517673 2764 raft_consensus.cc:482] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:48.517810 2764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:48.518872 2764 raft_consensus.cc:504] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b990deab46154acda7b3b3860d648935" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42211 } }
I20260812 07:06:48.518957 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:48.519295 2764 leader_election.cc:302] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b990deab46154acda7b3b3860d648935; no voters:
I20260812 07:06:48.519629 2764 leader_election.cc:288] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:48.519714 2777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:48.519973 2777 raft_consensus.cc:686] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [term 1 LEADER]: Becoming Leader. State: Replica: b990deab46154acda7b3b3860d648935, State: Running, Role: LEADER
I20260812 07:06:48.520319 2777 consensus_queue.cc:237] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b990deab46154acda7b3b3860d648935" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42211 } }
I20260812 07:06:48.520954 2764 sys_catalog.cc:564] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:48.521981 2781 sys_catalog.cc:455] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b990deab46154acda7b3b3860d648935. Latest consensus state: current_term: 1 leader_uuid: "b990deab46154acda7b3b3860d648935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b990deab46154acda7b3b3860d648935" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42211 } } }
I20260812 07:06:48.522109 2781 sys_catalog.cc:458] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:48.522424 2780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b990deab46154acda7b3b3860d648935" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b990deab46154acda7b3b3860d648935" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42211 } } }
I20260812 07:06:48.522521 2780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:48.522827 2800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:48.526196 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:48.521240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39657 (local address 127.27.110.254:42211)
0812 07:06:48.521362 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:48.521366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:48.521875 (+ 509us) server_negotiation.cc:407] Connection header received
0812 07:06:48.522008 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:48.522014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:48.522065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:48.522154 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:48.522459 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:48.523271 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:48.524395 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:48.524724 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:48.524844 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:48.525705 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:48.525709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:48.525713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:48.525729 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:48.525756 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:48.525772 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:48.525892 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:48.526018 (+ 126us) server_negotiation.cc:299] Negotiation successful
0812 07:06:48.526051 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:06:48.526537 2800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:48.531926 2800 catalog_manager.cc:1347] Generated new cluster ID: 4417e72eee2e405584d77d4c13140c49
I20260812 07:06:48.531997 2800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:48.549084 2800 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:48.550381 2800 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:48.554728 2800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b990deab46154acda7b3b3860d648935: Generated new TSK 0
I20260812 07:06:48.555429 2800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:48.562798 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2608
W20260812 07:06:48.578562 2579 connection.cc:535] client connection to 127.27.110.254:42211 recv error: Network error: failed to read from TLS socket (remote: 127.27.110.254:42211): Connection reset by peer (error 104)
2026-08-12T07:06:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1678 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:48 dist-test-slave-w4v5 krb5kdc[2817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:48 dist-test-slave-w4v5 krb5kdc[2817](info): set up 2 sockets
Aug 12 07:06:48 dist-test-slave-w4v5 krb5kdc[2817](info): commencing operation
krb5kdc: starting...
W20260812 07:06:49.867650 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:49 dist-test-slave-w4v5 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:49.994244 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43397
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:50.106606 2848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:50.106907 2848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:50.106995 2848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:50.110994 2848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:50.111088 2848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:50.111155 2848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:50.111212 2848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:50.111249 2848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:50.116571 2848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:43397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:50.117609 2848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:50.118659 2848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:50.127166 2854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:50.127164 2855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:50.127161 2857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:50.127382 2848 server_base.cc:1034] running on GCE node
I20260812 07:06:50.128118 2848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:50.129185 2848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:50.130347 2848 hybrid_clock.cc:648] HybridClock initialized: now 1786518410130330 us; error 30 us; skew 500 ppm
Aug 12 07:06:50 dist-test-slave-w4v5 krb5kdc[2817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518410, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:50.133163 2848 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:50.134011 2848 webserver.cc:460] Webserver started at http://127.27.110.254:33881/ using document root <none> and password file <none>
I20260812 07:06:50.134562 2848 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:50.134656 2848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:50.134886 2848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:50.136933 2848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "74637f793a4b4c609f1d9e53bcb6caac"
format_stamp: "Formatted at 2026-08-12 07:06:50 on dist-test-slave-w4v5"
server_key: "ec9d04256f3e9e49c9ed2ee0935c0e81"
server_key_iv: "666e4b51d72a0817fbfb6ef58f00b25f"
server_key_version: "encryptionkey@0"
I20260812 07:06:50.137498 2848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "74637f793a4b4c609f1d9e53bcb6caac"
format_stamp: "Formatted at 2026-08-12 07:06:50 on dist-test-slave-w4v5"
server_key: "ec9d04256f3e9e49c9ed2ee0935c0e81"
server_key_iv: "666e4b51d72a0817fbfb6ef58f00b25f"
server_key_version: "encryptionkey@0"
I20260812 07:06:50.141181 2848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:06:50.143617 2868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:50.144863 2848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:50.145015 2848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "74637f793a4b4c609f1d9e53bcb6caac"
format_stamp: "Formatted at 2026-08-12 07:06:50 on dist-test-slave-w4v5"
server_key: "ec9d04256f3e9e49c9ed2ee0935c0e81"
server_key_iv: "666e4b51d72a0817fbfb6ef58f00b25f"
server_key_version: "encryptionkey@0"
I20260812 07:06:50.145128 2848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:50.149734 2848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:50.152660 2848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:50.152853 2848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:50.159888 2848 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:43397
I20260812 07:06:50.159916 2987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:43397 every 8 connection(s)
I20260812 07:06:50.160815 2848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:50.163552 2990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:50.168802 2990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac: Bootstrap starting.
I20260812 07:06:50.170440 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 2848
I20260812 07:06:50.170624 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:50.170909 28091 external_mini_cluster.cc:1428] Setting key c6b72e0f4514b463e3c704cab97624ab
I20260812 07:06:50.171305 2990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:50.172256 2990 log.cc:824] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:50.174322 2990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac: No bootstrap required, opened a new log
I20260812 07:06:50.176065 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:50.172097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37653 (local address 127.27.110.254:43397)
0812 07:06:50.172493 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:50.172509 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:50.172550 (+ 41us) server_negotiation.cc:407] Connection header received
0812 07:06:50.173317 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:50.173337 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:50.173613 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:50.173891 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:50.174818 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:50.174824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:50.174841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:50.174853 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:50.174916 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:50.174935 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:50.175022 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:50.175202 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 07:06:50.175372 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":126,"threads_started":1}
I20260812 07:06:50.178560 2990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74637f793a4b4c609f1d9e53bcb6caac" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43397 } }
I20260812 07:06:50.178846 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:50.178872 2990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:50.178956 2990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74637f793a4b4c609f1d9e53bcb6caac, State: Initialized, Role: FOLLOWER
I20260812 07:06:50.179456 2990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74637f793a4b4c609f1d9e53bcb6caac" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43397 } }
I20260812 07:06:50.179611 2990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:50.179702 2990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:50.179803 2990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:50.180851 2990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74637f793a4b4c609f1d9e53bcb6caac" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43397 } }
I20260812 07:06:50.181241 2990 leader_election.cc:302] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74637f793a4b4c609f1d9e53bcb6caac; no voters:
I20260812 07:06:50.181598 2990 leader_election.cc:288] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:50.181685 3009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:50.181929 3009 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [term 1 LEADER]: Becoming Leader. State: Replica: 74637f793a4b4c609f1d9e53bcb6caac, State: Running, Role: LEADER
I20260812 07:06:50.182322 3009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74637f793a4b4c609f1d9e53bcb6caac" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43397 } }
I20260812 07:06:50.182922 2990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:50.182964 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:50.181067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49257 (local address 127.27.110.254:43397)
0812 07:06:50.181197 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:50.181201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:50.181411 (+ 210us) server_negotiation.cc:407] Connection header received
0812 07:06:50.181573 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:50.181577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:50.181614 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:50.181692 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:50.182558 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:50.182562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:50.182564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:50.182575 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:50.182601 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:50.182614 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:50.182693 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:50.182822 (+ 129us) server_negotiation.cc:299] Negotiation successful
0812 07:06:50.182858 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:06:50.183669 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74637f793a4b4c609f1d9e53bcb6caac. Latest consensus state: current_term: 1 leader_uuid: "74637f793a4b4c609f1d9e53bcb6caac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74637f793a4b4c609f1d9e53bcb6caac" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43397 } } }
I20260812 07:06:50.183869 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:50.184283 3022 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:50.184486 3010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74637f793a4b4c609f1d9e53bcb6caac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74637f793a4b4c609f1d9e53bcb6caac" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 43397 } } }
I20260812 07:06:50.184664 3010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:50.187470 3022 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:50.192497 3022 catalog_manager.cc:1347] Generated new cluster ID: e7fa5047861f42039330f8b705f0d734
I20260812 07:06:50.192559 3022 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:50.200379 3022 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:50.200681 3022 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:50.209234 3022 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74637f793a4b4c609f1d9e53bcb6caac: Generated new TSK 0
I20260812 07:06:50.209826 3022 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:50.218819 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 2848
2026-08-12T07:06:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1653 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:50 dist-test-slave-w4v5 krb5kdc[3042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:50 dist-test-slave-w4v5 krb5kdc[3042](info): set up 2 sockets
Aug 12 07:06:50 dist-test-slave-w4v5 krb5kdc[3042](info): commencing operation
krb5kdc: starting...
W20260812 07:06:51.550612 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:51 dist-test-slave-w4v5 krb5kdc[3042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:51.685964 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44139
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:35077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:44139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:51.801425 3074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:51.801712 3074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:51.801755 3074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:51.805859 3074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:51.805918 3074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:51.805938 3074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:51.805954 3074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:51.805969 3074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:51.811434 3074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:35077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:44139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:44139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:51.812539 3074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:51.813656 3074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:51.822939 3080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:51.822903 3081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:51.823158 3074 server_base.cc:1034] running on GCE node
W20260812 07:06:51.822940 3089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:51.824154 3074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:51.825322 3074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:51.826561 3074 hybrid_clock.cc:648] HybridClock initialized: now 1786518411826545 us; error 27 us; skew 500 ppm
Aug 12 07:06:51 dist-test-slave-w4v5 krb5kdc[3042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:51.834131 3074 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:51.835448 3074 webserver.cc:460] Webserver started at http://127.27.110.254:43019/ using document root <none> and password file <none>
I20260812 07:06:51.836063 3074 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:51.836164 3074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:51.836401 3074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:51.838562 3074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "3c9012c08d6c40cbbed86f0ff89667a3"
format_stamp: "Formatted at 2026-08-12 07:06:51 on dist-test-slave-w4v5"
server_key: "4a4888a6ad10b53523bb35d1fa63149c"
server_key_iv: "f42a109c685382eb05ea7e2d716d04f0"
server_key_version: "encryptionkey@0"
I20260812 07:06:51.839210 3074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "3c9012c08d6c40cbbed86f0ff89667a3"
format_stamp: "Formatted at 2026-08-12 07:06:51 on dist-test-slave-w4v5"
server_key: "4a4888a6ad10b53523bb35d1fa63149c"
server_key_iv: "f42a109c685382eb05ea7e2d716d04f0"
server_key_version: "encryptionkey@0"
I20260812 07:06:51.843001 3074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 07:06:51.845717 3100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:51.847055 3074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 07:06:51.847271 3074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "3c9012c08d6c40cbbed86f0ff89667a3"
format_stamp: "Formatted at 2026-08-12 07:06:51 on dist-test-slave-w4v5"
server_key: "4a4888a6ad10b53523bb35d1fa63149c"
server_key_iv: "f42a109c685382eb05ea7e2d716d04f0"
server_key_version: "encryptionkey@0"
I20260812 07:06:51.847388 3074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:51.858103 3074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:51.861389 3074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:51.861600 3074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:51.868875 3074 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:44139
I20260812 07:06:51.868903 3218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:44139 every 8 connection(s)
I20260812 07:06:51.869987 3074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:51.872756 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3074
I20260812 07:06:51.872833 3219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:51.872920 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:51.873255 28091 external_mini_cluster.cc:1428] Setting key 6062a28c873a9f1f09911ffbd0493eb6
I20260812 07:06:51.879317 3219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3: Bootstrap starting.
I20260812 07:06:51.881631 3219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3: Neither blocks nor log segments found. Creating new log.
Aug 12 07:06:51 dist-test-slave-w4v5 krb5kdc[3042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:51.882589 3219 log.cc:824] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:51.884907 3219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3: No bootstrap required, opened a new log
I20260812 07:06:51.887926 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:51.874747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45631 (local address 127.27.110.254:44139)
0812 07:06:51.875244 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:51.875260 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:51.875296 (+ 36us) server_negotiation.cc:407] Connection header received
0812 07:06:51.875996 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:51.876013 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:51.876345 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:51.876724 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:51.877527 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:51.878521 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:51.879524 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:51.880034 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:51.880226 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:51.882963 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:51.882968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:51.882981 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:51.883034 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:51.884974 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:51.885463 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:51.885468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:51.885472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:51.885588 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:51.885908 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:51.885911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:51.885912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:51.886316 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:51.886535 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:51.886840 (+ 305us) server_negotiation.cc:299] Negotiation successful
0812 07:06:51.887200 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":139,"threads_started":1}
I20260812 07:06:51.888557 3219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44139 } }
I20260812 07:06:51.888808 3219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:51.888887 3219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9012c08d6c40cbbed86f0ff89667a3, State: Initialized, Role: FOLLOWER
I20260812 07:06:51.889326 3219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44139 } }
I20260812 07:06:51.889451 3219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:51.889564 3219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:51.889693 3219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:51.890776 3219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44139 } }
I20260812 07:06:51.890985 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:51.891183 3219 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9012c08d6c40cbbed86f0ff89667a3; no voters:
I20260812 07:06:51.891595 3219 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:51.891677 3228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:51.891878 3228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9012c08d6c40cbbed86f0ff89667a3, State: Running, Role: LEADER
I20260812 07:06:51.892233 3228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44139 } }
I20260812 07:06:51.892901 3219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:51.894095 3230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9012c08d6c40cbbed86f0ff89667a3. Latest consensus state: current_term: 1 leader_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44139 } } }
I20260812 07:06:51.894094 3229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9012c08d6c40cbbed86f0ff89667a3" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 44139 } } }
I20260812 07:06:51.894261 3230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:51.894261 3229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:51.894691 3252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:51.897948 3252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:51.903308 3224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:51.893843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58043 (local address 127.27.110.254:44139)
0812 07:06:51.893964 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:51.893970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:51.895158 (+ 1188us) server_negotiation.cc:407] Connection header received
0812 07:06:51.895279 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:51.895285 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:51.895336 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:51.895430 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:51.896240 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:51.896753 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:51.897944 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:51.898177 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:51.898286 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:51.900048 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:51.900051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:51.900054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:51.900081 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:51.901534 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:51.901954 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:51.901962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:51.901963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:51.902043 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:51.902312 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:51.902316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:51.902317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:51.902531 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:51.902662 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:51.902982 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 07:06:51.903120 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260812 07:06:51.904222 3252 catalog_manager.cc:1347] Generated new cluster ID: 638d8e11f1854964893c98bcfda152f8
I20260812 07:06:51.904289 3252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:51.921525 3252 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:51.922446 3252 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:51.926937 3252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9012c08d6c40cbbed86f0ff89667a3: Generated new TSK 0
I20260812 07:06:51.927618 3252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:51.931862 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3074
2026-08-12T07:06:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1714 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:52 dist-test-slave-w4v5 krb5kdc[3277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:52 dist-test-slave-w4v5 krb5kdc[3277](info): set up 2 sockets
Aug 12 07:06:52 dist-test-slave-w4v5 krb5kdc[3277](info): commencing operation
krb5kdc: starting...
W20260812 07:06:53.258003 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:53 dist-test-slave-w4v5 krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:53.390358 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42209
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:42209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:53.503407 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:53.503662 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:53.503703 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:53.507678 3302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:53.507732 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:53.507751 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:53.507767 3302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:53.507782 3302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:53.512982 3302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:42209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:42209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:53.514022 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:53.514950 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:53.523928 3319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:53.523937 3315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:53.523948 3317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:53.524286 3302 server_base.cc:1034] running on GCE node
I20260812 07:06:53.524907 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:53.525952 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:53.527176 3302 hybrid_clock.cc:648] HybridClock initialized: now 1786518413527158 us; error 37 us; skew 500 ppm
Aug 12 07:06:53 dist-test-slave-w4v5 krb5kdc[3277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:53.530253 3302 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:53.531287 3302 webserver.cc:460] Webserver started at http://127.27.110.254:40793/ using document root <none> and password file <none>
I20260812 07:06:53.531855 3302 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:53.531917 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:53.532138 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:53.534158 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "ba99c1f071804b788523084c4e4538cd"
format_stamp: "Formatted at 2026-08-12 07:06:53 on dist-test-slave-w4v5"
server_key: "eb2c9ceaf74695a191da929e8ea2a8dd"
server_key_iv: "447c4c4fc30caa48fe99799eafc7e8fd"
server_key_version: "encryptionkey@0"
I20260812 07:06:53.534723 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "ba99c1f071804b788523084c4e4538cd"
format_stamp: "Formatted at 2026-08-12 07:06:53 on dist-test-slave-w4v5"
server_key: "eb2c9ceaf74695a191da929e8ea2a8dd"
server_key_iv: "447c4c4fc30caa48fe99799eafc7e8fd"
server_key_version: "encryptionkey@0"
I20260812 07:06:53.538455 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 07:06:53.540889 3347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:53.542135 3302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 07:06:53.542276 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "ba99c1f071804b788523084c4e4538cd"
format_stamp: "Formatted at 2026-08-12 07:06:53 on dist-test-slave-w4v5"
server_key: "eb2c9ceaf74695a191da929e8ea2a8dd"
server_key_iv: "447c4c4fc30caa48fe99799eafc7e8fd"
server_key_version: "encryptionkey@0"
I20260812 07:06:53.542388 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:53.555968 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:53.558876 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:53.559155 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:53.566880 3302 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:42209
I20260812 07:06:53.566907 3432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:42209 every 8 connection(s)
I20260812 07:06:53.567916 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:53.570715 3435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:53.576061 3435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd: Bootstrap starting.
I20260812 07:06:53.576720 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3302
I20260812 07:06:53.576875 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:53.577167 28091 external_mini_cluster.cc:1428] Setting key c106b6c0dd6cbf8bbbf0b8b4a48882f7
I20260812 07:06:53.578940 3435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:53.579860 3435 log.cc:824] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:53.582036 3435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd: No bootstrap required, opened a new log
I20260812 07:06:53.586337 3435 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba99c1f071804b788523084c4e4538cd" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42209 } }
I20260812 07:06:53.586653 3435 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:53.586732 3435 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba99c1f071804b788523084c4e4538cd, State: Initialized, Role: FOLLOWER
I20260812 07:06:53.587218 3435 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba99c1f071804b788523084c4e4538cd" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42209 } }
I20260812 07:06:53.587373 3435 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:53.587433 3435 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:53.587528 3435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:53.587663 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:53.578525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54277 (local address 127.27.110.254:42209)
0812 07:06:53.578992 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:53.579014 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:53.579057 (+ 43us) server_negotiation.cc:407] Connection header received
0812 07:06:53.579916 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:53.579940 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:53.580374 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:53.580717 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:53.581119 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:53.582545 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:53.583721 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:53.584425 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:53.584625 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:53.585615 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:53.585621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:53.585639 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:53.586135 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:53.586202 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:53.586233 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:53.586333 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:53.586476 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 07:06:53.586684 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260812 07:06:53.588622 3435 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba99c1f071804b788523084c4e4538cd" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42209 } }
I20260812 07:06:53.588970 3435 leader_election.cc:302] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba99c1f071804b788523084c4e4538cd; no voters:
I20260812 07:06:53.589274 3435 leader_election.cc:288] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:53.589378 3440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:53.589610 3440 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [term 1 LEADER]: Becoming Leader. State: Replica: ba99c1f071804b788523084c4e4538cd, State: Running, Role: LEADER
I20260812 07:06:53.589942 3440 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba99c1f071804b788523084c4e4538cd" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42209 } }
I20260812 07:06:53.590544 3435 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:53.590685 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:53.591223 3440 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba99c1f071804b788523084c4e4538cd. Latest consensus state: current_term: 1 leader_uuid: "ba99c1f071804b788523084c4e4538cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba99c1f071804b788523084c4e4538cd" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42209 } } }
I20260812 07:06:53.591358 3440 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:53.591751 3448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:53.591789 3441 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba99c1f071804b788523084c4e4538cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba99c1f071804b788523084c4e4538cd" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 42209 } } }
I20260812 07:06:53.591974 3441 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:53.595063 3448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:53.599243 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:53.593156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54185 (local address 127.27.110.254:42209)
0812 07:06:53.593262 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:53.593266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:53.594350 (+ 1084us) server_negotiation.cc:407] Connection header received
0812 07:06:53.594750 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:53.594755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:53.594806 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:53.594893 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:53.595264 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:53.595993 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:53.597360 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:53.597707 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:53.597816 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:53.598706 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:53.598710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:53.598713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:53.598727 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:53.598755 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:53.598769 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:53.598849 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:53.598990 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 07:06:53.599044 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 07:06:53.601275 3448 catalog_manager.cc:1347] Generated new cluster ID: f27c94fa908e4ee4ae0e5b7aa2e4ecfe
I20260812 07:06:53.601342 3448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:53.611743 3448 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:53.612628 3448 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:53.622471 3448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba99c1f071804b788523084c4e4538cd: Generated new TSK 0
I20260812 07:06:53.623245 3448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:53.627053 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3302
2026-08-12T07:06:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1694 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:53 dist-test-slave-w4v5 krb5kdc[3490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:53 dist-test-slave-w4v5 krb5kdc[3490](info): set up 2 sockets
Aug 12 07:06:53 dist-test-slave-w4v5 krb5kdc[3490](info): commencing operation
krb5kdc: starting...
W20260812 07:06:54.940838 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.285s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:55 dist-test-slave-w4v5 krb5kdc[3490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:55.069676 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39479
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:40493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:39479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:55.181229 3518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:55.181499 3518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:55.181542 3518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:55.185546 3518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:55.185607 3518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:55.185627 3518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:55.185644 3518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:55.185708 3518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:55.190938 3518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:40493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:39479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:55.192026 3518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:55.192997 3518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:55.201839 3530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:55.202003 3518 server_base.cc:1034] running on GCE node
W20260812 07:06:55.201839 3529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:55.201879 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:55.202694 3518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:55.203821 3518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:55.204979 3518 hybrid_clock.cc:648] HybridClock initialized: now 1786518415204949 us; error 43 us; skew 500 ppm
Aug 12 07:06:55 dist-test-slave-w4v5 krb5kdc[3490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518415, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:55.207788 3518 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:55.208765 3518 webserver.cc:460] Webserver started at http://127.27.110.254:33857/ using document root <none> and password file <none>
I20260812 07:06:55.209350 3518 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:55.209450 3518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:55.209708 3518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:55.211992 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "17af95de45144d8796c170c009231228"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-w4v5"
server_key: "e4571c4acac15feae7a26d6fe81128f0"
server_key_iv: "15b3e04570dccdb7121857d42a328a32"
server_key_version: "encryptionkey@0"
I20260812 07:06:55.212611 3518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "17af95de45144d8796c170c009231228"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-w4v5"
server_key: "e4571c4acac15feae7a26d6fe81128f0"
server_key_iv: "15b3e04570dccdb7121857d42a328a32"
server_key_version: "encryptionkey@0"
I20260812 07:06:55.216625 3518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 07:06:55.219275 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:55.220577 3518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:55.220733 3518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "17af95de45144d8796c170c009231228"
format_stamp: "Formatted at 2026-08-12 07:06:55 on dist-test-slave-w4v5"
server_key: "e4571c4acac15feae7a26d6fe81128f0"
server_key_iv: "15b3e04570dccdb7121857d42a328a32"
server_key_version: "encryptionkey@0"
I20260812 07:06:55.220852 3518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:55.225911 3518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:55.228981 3518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:55.229180 3518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:55.237196 3518 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:39479
I20260812 07:06:55.237231 3654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:39479 every 8 connection(s)
I20260812 07:06:55.238186 3518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:55.241072 3655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:55.245754 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3518
I20260812 07:06:55.245903 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:55.246229 28091 external_mini_cluster.cc:1428] Setting key ce7d3660e0eb75c0cd884745c23b02da
I20260812 07:06:55.247355 3655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228: Bootstrap starting.
I20260812 07:06:55.249764 3655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:55.250696 3655 log.cc:824] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:55.252010 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.247786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39163 (local address 127.27.110.254:39479)
0812 07:06:55.248253 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.248268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.248306 (+ 38us) server_negotiation.cc:407] Connection header received
0812 07:06:55.249038 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.249058 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.249356 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.249730 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.250582 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.250586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.250599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:55.250610 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.250667 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:55.250693 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.250816 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.250926 (+ 110us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.251139 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":143,"threads_started":1}
I20260812 07:06:55.252827 3655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228: No bootstrap required, opened a new log
I20260812 07:06:55.254832 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:55.256865 3655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17af95de45144d8796c170c009231228" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39479 } }
I20260812 07:06:55.257164 3655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 07:06:55.257231 3655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17af95de45144d8796c170c009231228, State: Initialized, Role: FOLLOWER
I20260812 07:06:55.257741 3655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17af95de45144d8796c170c009231228" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39479 } }
I20260812 07:06:55.257963 3655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 07:06:55.258039 3655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 07:06:55.258145 3655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:55.258877 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:55.256977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59205 (local address 127.27.110.254:39479)
0812 07:06:55.257091 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:55.257094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:55.257367 (+ 273us) server_negotiation.cc:407] Connection header received
0812 07:06:55.257494 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:55.257497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:55.257535 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:55.257616 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:55.258501 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:55.258504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:55.258506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 07:06:55.258516 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:55.258541 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 07:06:55.258551 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:55.258620 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:55.258721 (+ 101us) server_negotiation.cc:299] Negotiation successful
0812 07:06:55.258751 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 07:06:55.259405 3655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17af95de45144d8796c170c009231228" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39479 } }
I20260812 07:06:55.259773 3655 leader_election.cc:302] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17af95de45144d8796c170c009231228; no voters:
I20260812 07:06:55.260138 3655 leader_election.cc:288] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 07:06:55.260236 3676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:55.260491 3676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [term 1 LEADER]: Becoming Leader. State: Replica: 17af95de45144d8796c170c009231228, State: Running, Role: LEADER
I20260812 07:06:55.260860 3676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17af95de45144d8796c170c009231228" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39479 } }
I20260812 07:06:55.261332 3655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:55.262686 3679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17af95de45144d8796c170c009231228. Latest consensus state: current_term: 1 leader_uuid: "17af95de45144d8796c170c009231228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17af95de45144d8796c170c009231228" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39479 } } }
I20260812 07:06:55.262640 3678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17af95de45144d8796c170c009231228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17af95de45144d8796c170c009231228" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 39479 } } }
I20260812 07:06:55.262854 3678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:55.262854 3679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:55.263340 3693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:55.266572 3693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:55.272291 3693 catalog_manager.cc:1347] Generated new cluster ID: 48389fabacff4a5daf866f0ed4495101
I20260812 07:06:55.272360 3693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:55.289721 3693 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:55.290071 3693 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:55.293947 3693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17af95de45144d8796c170c009231228: Generated new TSK 0
I20260812 07:06:55.294746 3693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 07:06:55.308230 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3518
2026-08-12T07:06:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1683 ms)
[----------] 18 tests from AuthTokenIssuingTest (30583 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:55 dist-test-slave-w4v5 krb5kdc[3715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:55 dist-test-slave-w4v5 krb5kdc[3715](info): set up 2 sockets
Aug 12 07:06:55 dist-test-slave-w4v5 krb5kdc[3715](info): commencing operation
krb5kdc: starting...
W20260812 07:06:56.618759 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.279s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:06:56 dist-test-slave-w4v5 krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:06:56.758229 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36275
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36275,127.27.110.253:46165,127.27.110.252:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:06:56.876546 3740 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:06:56.876888 3740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:56.876965 3740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:56.877048 3740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:06:56.881049 3740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:56.881147 3740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:56.881186 3740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:56.881222 3740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:56.881251 3740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:56.886463 3740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36275,127.27.110.253:46165,127.27.110.252:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:56.887568 3740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:56.888618 3740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:56.897725 3754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:56.897696 3755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:56.897697 3740 server_base.cc:1034] running on GCE node
W20260812 07:06:56.897709 3757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:56.898589 3740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:56.899782 3740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:56.900961 3740 hybrid_clock.cc:648] HybridClock initialized: now 1786518416900931 us; error 41 us; skew 500 ppm
Aug 12 07:06:56 dist-test-slave-w4v5 krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:56.908367 3740 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:06:56.909469 3740 webserver.cc:460] Webserver started at http://127.27.110.254:43133/ using document root <none> and password file <none>
I20260812 07:06:56.910090 3740 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:56.910159 3740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:56.910399 3740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:56.912612 3740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "e80b2b8ef2384386bb492373f1dcdea4"
format_stamp: "Formatted at 2026-08-12 07:06:56 on dist-test-slave-w4v5"
server_key: "1e939b39bf485634e45f7ab1d10020e5"
server_key_iv: "c33994d938a224c84804335f0bfff3e2"
server_key_version: "encryptionkey@0"
I20260812 07:06:56.913203 3740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "e80b2b8ef2384386bb492373f1dcdea4"
format_stamp: "Formatted at 2026-08-12 07:06:56 on dist-test-slave-w4v5"
server_key: "1e939b39bf485634e45f7ab1d10020e5"
server_key_iv: "c33994d938a224c84804335f0bfff3e2"
server_key_version: "encryptionkey@0"
I20260812 07:06:56.917135 3740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:06:56.919755 3770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:56.921169 3740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:56.921337 3740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "e80b2b8ef2384386bb492373f1dcdea4"
format_stamp: "Formatted at 2026-08-12 07:06:56 on dist-test-slave-w4v5"
server_key: "1e939b39bf485634e45f7ab1d10020e5"
server_key_iv: "c33994d938a224c84804335f0bfff3e2"
server_key_version: "encryptionkey@0"
I20260812 07:06:56.921463 3740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:56.940579 3740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:56.944443 3740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:56.944799 3740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:56.953045 3740 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36275
I20260812 07:06:56.953344 3864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36275 every 8 connection(s)
I20260812 07:06:56.954196 3740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:06:56.957348 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3740
I20260812 07:06:56.957441 3865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:56.957541 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:06:56.957836 28091 external_mini_cluster.cc:1428] Setting key 34b9b11395627c1ece75509bfb2a0acf
I20260812 07:06:56.961520 3865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.253@KRBTEST.COM" created.
Aug 12 07:06:56 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:56.977813 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:56.963927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36275 (local address 127.0.0.1:60928)
0812 07:06:56.964481 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:56.964545 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:56.965518 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:56.966439 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:56.966443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:56.966817 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:56.967168 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:56.967193 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:56.968507 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:56.968511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:56.969262 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:56.969272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:56.969894 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:56.969897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:56.970096 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:56.970777 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:56.970793 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:56.972586 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:56.974604 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:56.974613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:56.974623 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:56.974949 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:56.975332 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:56.975336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:56.975338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:56.975461 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:56.976062 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:56.976069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:56.976211 (+ 142us) client_negotiation.cc:769] Sending connection context
0812 07:06:56.976378 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 07:06:56.976745 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"thread_start_us":175,"threads_started":1}
I20260812 07:06:56.977813 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:56.964886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.27.110.254:36275)
0812 07:06:56.965253 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:56.965267 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:56.965320 (+ 53us) server_negotiation.cc:407] Connection header received
0812 07:06:56.965872 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:56.965882 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:56.966153 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:56.966406 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:56.967343 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:56.968335 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:56.969369 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:56.969730 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:56.969860 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:56.972746 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:56.972752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:56.972762 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:56.972782 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:56.974472 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:56.975063 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:56.975069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:56.975099 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:56.975233 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:56.975556 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:56.975561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:56.975562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:56.975932 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:56.976076 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:56.976463 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 07:06:56.976797 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260812 07:06:56.980372 3865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:56.981204 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:56.980931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58614)
0812 07:06:56.981045 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:56.981126 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 07:06:56.981451 3774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
I20260812 07:06:56.982385 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:56.982186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58618)
0812 07:06:56.982288 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:56.982332 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 07:06:56.982612 3865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:46165: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab.
Entry for principal kudu/127.27.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab.
I20260812 07:06:56.986263 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:46165
--webserver_interface=127.27.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab
--principal=kudu/127.27.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36275,127.27.110.253:46165,127.27.110.252:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:06:57.031880 3865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } attempt: 1
I20260812 07:06:57.032809 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.032481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58626)
0812 07:06:57.032641 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.032712 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 07:06:57.033613 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.033381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58634)
0812 07:06:57.033496 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.033550 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 07:06:57.033788 3865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:46165: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
W20260812 07:06:57.101408 3879 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:06:57.101673 3879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:57.101696 3879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:57.101733 3879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 07:06:57.101979 3865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } attempt: 2
I20260812 07:06:57.102890 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.102565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58646)
0812 07:06:57.102744 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.102810 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 07:06:57.103617 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.103424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58660)
0812 07:06:57.103522 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.103565 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 07:06:57.103845 3865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:46165: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46165: connect: Connection refused (error 111)
W20260812 07:06:57.106324 3879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:57.106380 3879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:57.106400 3879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:57.106444 3879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:57.106477 3879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:57.112294 3879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36275,127.27.110.253:46165,127.27.110.252:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab
--principal=kudu/127.27.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:46165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:57.113399 3879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:57.114482 3879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:06:57.123104 3890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:57.123142 3894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:57.123273 3888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:57.123517 3879 server_base.cc:1034] running on GCE node
I20260812 07:06:57.124076 3879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:57.125278 3879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:57.126453 3879 hybrid_clock.cc:648] HybridClock initialized: now 1786518417126424 us; error 48 us; skew 500 ppm
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:57.129271 3879 init.cc:377] Logged in from keytab as kudu/127.27.110.253@KRBTEST.COM (short username kudu)
I20260812 07:06:57.130108 3879 webserver.cc:460] Webserver started at http://127.27.110.253:39489/ using document root <none> and password file <none>
I20260812 07:06:57.130652 3879 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:57.130723 3879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:57.130896 3879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:57.133098 3879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data/instance:
uuid: "1d7ff020dd6b4c529d9618c34d0f6d72"
format_stamp: "Formatted at 2026-08-12 07:06:57 on dist-test-slave-w4v5"
server_key: "a4092153a26cc950c9d0547b1f826d32"
server_key_iv: "d83625eea4429e3f983d69ed8f0f9e61"
server_key_version: "encryptionkey@0"
I20260812 07:06:57.133636 3879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal/instance:
uuid: "1d7ff020dd6b4c529d9618c34d0f6d72"
format_stamp: "Formatted at 2026-08-12 07:06:57 on dist-test-slave-w4v5"
server_key: "a4092153a26cc950c9d0547b1f826d32"
server_key_iv: "d83625eea4429e3f983d69ed8f0f9e61"
server_key_version: "encryptionkey@0"
I20260812 07:06:57.137508 3879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 07:06:57.140033 3901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:57.141270 3879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:06:57.141409 3879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal
uuid: "1d7ff020dd6b4c529d9618c34d0f6d72"
format_stamp: "Formatted at 2026-08-12 07:06:57 on dist-test-slave-w4v5"
server_key: "a4092153a26cc950c9d0547b1f826d32"
server_key_iv: "d83625eea4429e3f983d69ed8f0f9e61"
server_key_version: "encryptionkey@0"
I20260812 07:06:57.141516 3879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:57.164052 3879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:57.167582 3879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:57.167838 3879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:57.175271 3879 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.253:46165
I20260812 07:06:57.175558 3987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.253:46165 every 8 connection(s)
I20260812 07:06:57.176354 3879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
I20260812 07:06:57.179245 3990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:57.183355 3990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.184046 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 3879
I20260812 07:06:57.184178 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-1/wal/instance
I20260812 07:06:57.184509 28091 external_mini_cluster.cc:1428] Setting key 8e230b798846e37ae3fa7e5135a84718
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.110.252@KRBTEST.COM; defaulting to no policy
I20260812 07:06:57.195031 3865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.252@KRBTEST.COM" created.
I20260812 07:06:57.200137 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.185896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.27.110.254:36275)
0812 07:06:57.186075 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.186080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.187134 (+ 1054us) server_negotiation.cc:407] Connection header received
0812 07:06:57.188249 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.188252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.188309 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.188397 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.190147 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.190694 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.192023 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.192473 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.192595 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.195053 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.195056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.195058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.195132 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.197667 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.198294 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.198298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.198300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.198391 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.198749 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.198754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.198756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.198949 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.199060 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.199764 (+ 704us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.199916 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 07:06:57.201360 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.186355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36275 (local address 127.0.0.1:60932)
0812 07:06:57.186910 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.186978 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.187952 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.188618 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.188630 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.189199 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.189964 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.189995 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.190832 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.190838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.191858 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.191868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.192631 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.192634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.192778 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.193432 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.193449 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.194883 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.197799 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.197805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.197820 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.198188 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.198489 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.198493 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.198495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.198652 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.199059 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.199066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.199429 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.199635 (+ 206us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.199977 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":181,"threads_started":1}
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:06:57.203308 3990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.206310 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.195603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58676)
0812 07:06:57.195776 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.195794 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.195876 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.196389 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.196395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.196605 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.196777 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.196789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.198198 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.198204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.199163 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.199178 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.199959 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.199963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.200061 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.200512 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.200526 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.202299 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.204622 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.204626 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.204628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.204901 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.205208 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.205211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.205213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.205301 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.205763 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.205766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.205868 (+ 102us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.205981 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.206138 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260812 07:06:57.206403 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.195697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.27.110.253:46165)
0812 07:06:57.196030 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.196040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.196061 (+ 21us) server_negotiation.cc:407] Connection header received
0812 07:06:57.196094 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.196098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.196252 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.196362 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.196893 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.198047 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.199291 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.199820 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.200283 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.202454 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.202462 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.202475 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.202500 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.204493 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.205031 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.205037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.205039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.205130 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.205398 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.205403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.205404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.205652 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.205796 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.206081 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.206222 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":144,"threads_started":1}
I20260812 07:06:57.208870 3865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.209640 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.209396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36374)
0812 07:06:57.209514 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.209565 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:06:57.210408 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.210158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36388)
0812 07:06:57.210303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.210350 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260812 07:06:57.210601 3865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:32891: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
I20260812 07:06:57.213958 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.203905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58684)
0812 07:06:57.204090 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.204103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.204177 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.204712 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.204718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.204882 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.205018 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.205025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.206204 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.206207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.207227 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.207243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.207896 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.207899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.208047 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.208450 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.208464 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.210579 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.212498 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.212500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.212502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.212730 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.212988 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.212990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.212991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.213065 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.213441 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.213443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.213520 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.213602 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.213747 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":8}
I20260812 07:06:57.213971 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.203999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.27.110.253:46165)
0812 07:06:57.204446 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.204458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.204476 (+ 18us) server_negotiation.cc:407] Connection header received
0812 07:06:57.204523 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.204530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.204586 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.204685 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.205146 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.206048 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.207383 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.207772 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.207905 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.210804 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.210807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.210809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.210830 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.212395 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.212822 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.212825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.212826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.212905 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.213165 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.213168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.213170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.213359 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.213436 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.213712 (+ 276us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.213810 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":352,"thread_start_us":177,"threads_started":1}
I20260812 07:06:57.214622 3990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.215421 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.215189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36404)
0812 07:06:57.215294 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.215369 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 07:06:57.215767 3909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
I20260812 07:06:57.216733 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.216504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36410)
0812 07:06:57.216602 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.216659 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 07:06:57.216943 3990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:32891: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab.
Entry for principal kudu/127.27.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab.
I20260812 07:06:57.218367 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:32891
--webserver_interface=127.27.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab
--principal=kudu/127.27.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36275,127.27.110.253:46165,127.27.110.252:32891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:06:57.241801 3865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } attempt: 1
I20260812 07:06:57.242889 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.242548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36420)
0812 07:06:57.242730 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.242799 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:06:57.243772 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.243519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36422)
0812 07:06:57.243649 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.243714 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 07:06:57.243903 3865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:32891: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
I20260812 07:06:57.266254 3990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } attempt: 1
I20260812 07:06:57.267254 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.266888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36428)
0812 07:06:57.267048 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.267160 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:06:57.268077 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.267833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36434)
0812 07:06:57.267953 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.268004 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 07:06:57.268261 3990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:32891: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
I20260812 07:06:57.319115 3865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } attempt: 2
I20260812 07:06:57.320019 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.319706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36446)
0812 07:06:57.319871 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.319937 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:06:57.320811 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.320568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36454)
0812 07:06:57.320678 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.320753 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 07:06:57.320950 3865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:32891: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
W20260812 07:06:57.335568 4004 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:06:57.335875 4004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:06:57.335939 4004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:06:57.336014 4004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 07:06:57.336448 3990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } attempt: 2
I20260812 07:06:57.337184 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.336899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36458)
0812 07:06:57.337043 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.337110 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 07:06:57.337960 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.337731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36474)
0812 07:06:57.337845 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.337905 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 07:06:57.338169 3990 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:32891: Network error: Client connection negotiation failed: client connection to 127.27.110.252:32891: connect: Connection refused (error 111)
W20260812 07:06:57.340562 4004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:06:57.340611 4004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:06:57.340654 4004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:06:57.340698 4004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:06:57.340715 4004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:06:57.346056 4004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36275,127.27.110.253:46165,127.27.110.252:32891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab
--principal=kudu/127.27.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:32891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:06:57.347258 4004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:06:57.348299 4004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 07:06:57.356921 4004 server_base.cc:1034] running on GCE node
W20260812 07:06:57.356892 4021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:57.356956 4022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:06:57.356956 4027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:06:57.357717 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:06:57.358731 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:06:57.359890 4004 hybrid_clock.cc:648] HybridClock initialized: now 1786518417359878 us; error 26 us; skew 500 ppm
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:06:57.362547 4004 init.cc:377] Logged in from keytab as kudu/127.27.110.252@KRBTEST.COM (short username kudu)
I20260812 07:06:57.363526 4004 webserver.cc:460] Webserver started at http://127.27.110.252:45363/ using document root <none> and password file <none>
I20260812 07:06:57.364081 4004 fs_manager.cc:359] Metadata directory not provided
I20260812 07:06:57.364174 4004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:06:57.364406 4004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:06:57.366461 4004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data/instance:
uuid: "a0d2f4086b594abea00752ff3fd3638a"
format_stamp: "Formatted at 2026-08-12 07:06:57 on dist-test-slave-w4v5"
server_key: "5986c912473beb3b86c3aa61435c9bcd"
server_key_iv: "24d780705d4326420caad7d419d49ef2"
server_key_version: "encryptionkey@0"
I20260812 07:06:57.367039 4004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal/instance:
uuid: "a0d2f4086b594abea00752ff3fd3638a"
format_stamp: "Formatted at 2026-08-12 07:06:57 on dist-test-slave-w4v5"
server_key: "5986c912473beb3b86c3aa61435c9bcd"
server_key_iv: "24d780705d4326420caad7d419d49ef2"
server_key_version: "encryptionkey@0"
I20260812 07:06:57.370772 4004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:06:57.373201 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:06:57.374362 4004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260812 07:06:57.374501 4004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal
uuid: "a0d2f4086b594abea00752ff3fd3638a"
format_stamp: "Formatted at 2026-08-12 07:06:57 on dist-test-slave-w4v5"
server_key: "5986c912473beb3b86c3aa61435c9bcd"
server_key_iv: "24d780705d4326420caad7d419d49ef2"
server_key_version: "encryptionkey@0"
I20260812 07:06:57.374612 4004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:06:57.394701 4004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:06:57.397641 4004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:06:57.397857 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:06:57.405158 4004 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.252:32891
I20260812 07:06:57.405463 4128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.252:32891 every 8 connection(s)
I20260812 07:06:57.406162 4004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
I20260812 07:06:57.408943 4130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:06:57.412822 4130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.415977 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4004
I20260812 07:06:57.416107 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786518006756923-28091-0/minicluster-data/master-2/wal/instance
I20260812 07:06:57.416437 28091 external_mini_cluster.cc:1428] Setting key 73ace3386d11c111ace9804b6976b1e7
I20260812 07:06:57.420114 3865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } attempt: 3
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:06:57.429030 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.414676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.27.110.254:36275)
0812 07:06:57.414827 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.414831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.415688 (+ 857us) server_negotiation.cc:407] Connection header received
0812 07:06:57.416915 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.416920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.416983 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.417072 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.418434 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.419272 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.420750 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.421082 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.421154 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.424696 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.424700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.424703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.424729 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.426595 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.427254 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.427260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.427262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.427362 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.427775 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.427779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.427784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.428023 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.428135 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.428692 (+ 557us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.428826 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:06:57.429487 3990 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } attempt: 3
I20260812 07:06:57.430039 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.415022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36275 (local address 127.0.0.1:60942)
0812 07:06:57.415530 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.415573 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.416663 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.417474 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.417480 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.417832 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.418289 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.418301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.419394 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.419398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.420608 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.420617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.421360 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.421364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.421774 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.422624 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.422647 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.424538 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.426704 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.426714 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.426727 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.427131 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.427451 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.427454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.427456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.427645 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.428136 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.428144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.428403 (+ 259us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.428601 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.428993 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":123,"spinlock_wait_cycles":768,"thread_start_us":116,"threads_started":1}
I20260812 07:06:57.430356 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.417137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.27.110.252:32891)
0812 07:06:57.417431 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.417444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.417631 (+ 187us) server_negotiation.cc:407] Connection header received
0812 07:06:57.417805 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.417810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.417964 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.418128 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.418990 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.420359 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.421566 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.422063 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.422180 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.425563 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.425568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.425578 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.425624 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.427715 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.428278 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.428287 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.428291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.428411 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.428747 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.428750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.428752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.429031 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.429228 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.430021 (+ 793us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.430196 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":185,"spinlock_wait_cycles":24320,"thread_start_us":83,"threads_started":1}
I20260812 07:06:57.431169 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.420506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36488)
0812 07:06:57.420712 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.420727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.420809 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.421175 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.421181 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.421351 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.421457 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.421462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.422643 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.422662 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.423242 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.423273 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.423927 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.423931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.424033 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.424545 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.424566 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.426150 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.429127 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.429130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.429132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.429410 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.430051 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.430054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.430056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.430154 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.430639 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.430642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.430765 (+ 123us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.430846 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.430986 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 07:06:57.431885 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.420560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.27.110.252:32891)
0812 07:06:57.420980 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.420986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.420996 (+ 10us) server_negotiation.cc:407] Connection header received
0812 07:06:57.421020 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.421022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.421052 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.421149 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.421609 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.422521 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.423389 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.423795 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.423929 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.426287 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.426294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.426299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.426325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.429002 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.429828 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.429832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.429834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.429937 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.430276 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.430280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.430281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.430520 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.431464 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.431540 (+ 76us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.431720 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":370,"thread_start_us":118,"threads_started":1}
I20260812 07:06:57.432876 4130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.435424 3865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4: Bootstrap starting.
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:06:57.438470 3865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4: Neither blocks nor log segments found. Creating new log.
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:06:57.439452 3865 log.cc:824] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:57.440146 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.429991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36502)
0812 07:06:57.430285 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.430302 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.430380 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.431009 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.431013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.431246 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.431378 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.431386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.432378 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.432383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.433172 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.433186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.433757 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.433761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.433840 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.434367 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.434383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.435809 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.438302 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.438305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.438310 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.438569 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.438933 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.438938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.438943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.439051 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.439607 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.439610 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.439735 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.439843 (+ 108us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.439974 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"spinlock_wait_cycles":2432}
I20260812 07:06:57.440781 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.430551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.27.110.252:32891)
0812 07:06:57.430776 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.430781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.430794 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:06:57.430827 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.430830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.430866 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.430947 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.431482 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.432239 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.433330 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.433621 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.435937 (+ 2316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.436035 (+ 98us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.436038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.436040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.436057 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.438171 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.438704 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.438709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.438712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.438802 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.439205 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.439210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.439212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.439482 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.440315 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.440433 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.440545 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260812 07:06:57.443234 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.433356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46165 (local address 127.0.0.1:58690)
0812 07:06:57.433718 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.433732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.433843 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.434174 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.434180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.434363 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.434495 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.434503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.435582 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.435585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.436428 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.436439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.437137 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.437142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.437260 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.437647 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.437658 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.439564 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.441525 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.441530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.441532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.441795 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.442096 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.442100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.442104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.442208 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.442731 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.442734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.442840 (+ 106us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.442943 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.443068 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299}
I20260812 07:06:57.443703 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.433577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.27.110.253:46165)
0812 07:06:57.433696 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.433700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.433781 (+ 81us) server_negotiation.cc:407] Connection header received
0812 07:06:57.433944 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.433947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.433995 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.434041 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.434642 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.435451 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.436568 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.436915 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.436987 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.439739 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.439744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.439747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.439786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.441407 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.441889 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.441895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.441900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.441987 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.442318 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.442321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.442323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.442580 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.442682 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.443451 (+ 769us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.443562 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":1536}
I20260812 07:06:57.443794 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.433321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.27.110.254:36275)
0812 07:06:57.433606 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.433610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.433624 (+ 14us) server_negotiation.cc:407] Connection header received
0812 07:06:57.433892 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.433898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.433943 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.433991 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.434870 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.435590 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.436696 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.437005 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.437085 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.440041 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.440044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.440047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.440071 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.442006 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.442503 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.442508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.442509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.442597 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.442887 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.442891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.442893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.443066 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.443282 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.443495 (+ 213us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.443611 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260812 07:06:57.443970 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: Bootstrap starting.
I20260812 07:06:57.444661 4130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:06:57.446949 3990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:57.448168 3990 log.cc:824] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: Log is configured to *not* fsync() on all Append() calls
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:06:57.450295 3865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4: No bootstrap required, opened a new log
Aug 12 07:06:57 dist-test-slave-w4v5 krb5kdc[3715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:06:57.453053 3990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: No bootstrap required, opened a new log
I20260812 07:06:57.453466 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.445110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:32891 (local address 127.0.0.1:36516)
0812 07:06:57.445380 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 07:06:57.445393 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 07:06:57.445470 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:06:57.445789 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:06:57.445793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:06:57.445979 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:06:57.446083 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.446089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.446766 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.446771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.447530 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:06:57.447540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.447965 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.447968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:06:57.448067 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.448545 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:06:57.448560 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:06:57.449790 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:06:57.451707 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.451713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.451718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.451946 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.452228 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:06:57.452231 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:06:57.452233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:06:57.452314 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:06:57.452977 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:06:57.452979 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:06:57.453063 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 07:06:57.453155 (+ 92us) client_negotiation.cc:240] Negotiation successful
0812 07:06:57.453296 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":4}
I20260812 07:06:57.454090 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.445136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.27.110.252:32891)
0812 07:06:57.445242 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.445246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.445525 (+ 279us) server_negotiation.cc:407] Connection header received
0812 07:06:57.445559 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.445560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.445637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.446173 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.446236 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.446663 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.447654 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.447865 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.447965 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.449922 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.449926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.449928 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.449951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.451577 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.452053 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.452056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.452057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.452128 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.452636 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.452639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.452641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.452870 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.453772 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.453844 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.453954 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":31}
I20260812 07:06:57.455260 3865 raft_consensus.cc:348] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.455585 3865 raft_consensus.cc:729] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e80b2b8ef2384386bb492373f1dcdea4, State: Initialized, Role: FOLLOWER
I20260812 07:06:57.455886 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.445380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.27.110.253:46165)
0812 07:06:57.445491 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.445495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.445730 (+ 235us) server_negotiation.cc:407] Connection header received
0812 07:06:57.445920 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.445925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.445972 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.446049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.446841 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.447589 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.448647 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.448942 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.449018 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.452011 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.452014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.452017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.452045 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.453745 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.454246 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.454250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.454251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.454340 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.454790 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.454794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.454796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.455003 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.455247 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.455615 (+ 368us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.455747 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260812 07:06:57.456202 3865 consensus_queue.cc:260] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.456872 4145 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.457059 4145 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:06:57.457309 4130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: Bootstrap starting.
I20260812 07:06:57.457830 3865 sys_catalog.cc:564] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:57.458051 3990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.458252 3990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7ff020dd6b4c529d9618c34d0f6d72, State: Initialized, Role: FOLLOWER
I20260812 07:06:57.458544 3990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.459100 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.459267 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:06:57.460078 3990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 07:06:57.463194 4175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:57.463299 4175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:57.463537 4130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: Neither blocks nor log segments found. Creating new log.
I20260812 07:06:57.464177 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:06:57.464697 4130 log.cc:824] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: Log is configured to *not* fsync() on all Append() calls
I20260812 07:06:57.466747 4130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: No bootstrap required, opened a new log
I20260812 07:06:57.467267 3817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e80b2b8ef2384386bb492373f1dcdea4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60946
I20260812 07:06:57.467528 3817 raft_consensus.cc:482] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 07:06:57.467636 3817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:57.468948 3817 raft_consensus.cc:504] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
W20260812 07:06:57.469777 4192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:06:57.469847 4192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:57.469830 3817 leader_election.cc:288] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [CANDIDATE]: Term 1 election: Requested vote from peers 1d7ff020dd6b4c529d9618c34d0f6d72 (127.27.110.253:46165), a0d2f4086b594abea00752ff3fd3638a (127.27.110.252:32891)
I20260812 07:06:57.467233 3740 master_runner.cc:186] Error getting master registration for 127.27.110.252:32891: OK, instance_id { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" instance_seqno: 1786518417403011 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:06:57.470877 4130 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.471205 4130 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d2f4086b594abea00752ff3fd3638a, State: Initialized, Role: FOLLOWER
I20260812 07:06:57.471591 4130 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.472884 4130 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:06:57.472934 4088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e80b2b8ef2384386bb492373f1dcdea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a0d2f4086b594abea00752ff3fd3638a"
I20260812 07:06:57.473151 4088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:57.471757 3879 master_runner.cc:186] Error getting master registration for 127.27.110.252:32891: OK, instance_id { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" instance_seqno: 1786518417403011 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:06:57.475014 3879 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:06:57.475178 4088 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e80b2b8ef2384386bb492373f1dcdea4 in term 1.
I20260812 07:06:57.475517 3944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e80b2b8ef2384386bb492373f1dcdea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72"
I20260812 07:06:57.475721 3944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:06:57.477023 3944 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e80b2b8ef2384386bb492373f1dcdea4 in term 1.
I20260812 07:06:57.475651 3775 leader_election.cc:302] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d2f4086b594abea00752ff3fd3638a, e80b2b8ef2384386bb492373f1dcdea4; no voters:
I20260812 07:06:57.478175 4145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:06:57.478402 4145 raft_consensus.cc:686] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [term 1 LEADER]: Becoming Leader. State: Replica: e80b2b8ef2384386bb492373f1dcdea4, State: Running, Role: LEADER
I20260812 07:06:57.478717 4145 consensus_queue.cc:237] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } }
I20260812 07:06:57.479494 4203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.479609 4203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: This master's current role is: FOLLOWER
W20260812 07:06:57.482039 4226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260812 07:06:57.482818 3740 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:06:57.483139 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80b2b8ef2384386bb492373f1dcdea4. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.483319 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: This master's current role is: LEADER
W20260812 07:06:57.487107 4226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:06:57.498355 4236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:06:57.499446 4236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:06:57.504335 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.483297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.27.110.252:32891)
0812 07:06:57.483423 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.483428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.487699 (+ 4271us) server_negotiation.cc:407] Connection header received
0812 07:06:57.487979 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.487983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.488030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.488085 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.493012 (+ 4927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.493784 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.497231 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.497581 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.497662 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.499889 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.499894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.499896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.499923 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.501665 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.502158 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.502162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.502164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.502249 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.502548 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.502551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.502553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.502765 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.502874 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.503791 (+ 917us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.503915 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 07:06:57.507352 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.482566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.27.110.254:36275)
0812 07:06:57.482675 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.482680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.487472 (+ 4792us) server_negotiation.cc:407] Connection header received
0812 07:06:57.489168 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.489171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.489222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.489275 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.492074 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.492857 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.496266 (+ 3409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.496606 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.496678 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.503823 (+ 7145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.503827 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.503830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.503856 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.505500 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.505998 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.506002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.506004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.506089 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.506379 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.506382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.506384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.506598 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.506709 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.507059 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.507202 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 07:06:57.512883 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:57.483557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.27.110.253:46165)
0812 07:06:57.483665 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.483669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.490458 (+ 6789us) server_negotiation.cc:407] Connection header received
0812 07:06:57.490598 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.490601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.490650 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.490701 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.497560 (+ 6859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.498648 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.502928 (+ 4280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:57.503286 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:57.503377 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:57.509166 (+ 5789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:06:57.509170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:06:57.509173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:06:57.509203 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:06:57.510981 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.511511 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.511515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.511517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.511604 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:06:57.511897 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:06:57.511900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:06:57.511902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:06:57.512116 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:06:57.512245 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:57.512596 (+ 351us) server_negotiation.cc:299] Negotiation successful
0812 07:06:57.512720 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:06:57.521708 4088 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [term 1 FOLLOWER]: Refusing update from remote peer e80b2b8ef2384386bb492373f1dcdea4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:57.522425 4227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:57.528613 4203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80b2b8ef2384386bb492373f1dcdea4. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.528745 4203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:06:57.530946 3944 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [term 1 FOLLOWER]: Refusing update from remote peer e80b2b8ef2384386bb492373f1dcdea4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:06:57.531718 4227 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:06:57.538602 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e80b2b8ef2384386bb492373f1dcdea4. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.538733 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:06:57.548359 4203 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.548496 4203 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:06:57.547024 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.551335 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:57.551585 4240 mvcc.cc:205] Tried to move back new op lower bound from 7317579438149709824 to 7317579437997916160. Current Snapshot: MvccSnapshot[applied={T|T < 7317579438149709824}]
I20260812 07:06:57.552935 4227 sys_catalog.cc:455] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.553041 4227 sys_catalog.cc:458] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4 [sys.catalog]: This master's current role is: LEADER
I20260812 07:06:57.554049 4156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e80b2b8ef2384386bb492373f1dcdea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e80b2b8ef2384386bb492373f1dcdea4" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36275 } } peers { permanent_uuid: "1d7ff020dd6b4c529d9618c34d0f6d72" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46165 } } peers { permanent_uuid: "a0d2f4086b594abea00752ff3fd3638a" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 32891 } } }
I20260812 07:06:57.554165 4156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:06:57.555172 4242 mvcc.cc:205] Tried to move back new op lower bound from 7317579438149709824 to 7317579437997916160. Current Snapshot: MvccSnapshot[applied={T|T < 7317579438149709824}]
I20260812 07:06:57.555413 4247 mvcc.cc:205] Tried to move back new op lower bound from 7317579438149709824 to 7317579437997916160. Current Snapshot: MvccSnapshot[applied={T|T < 7317579438149709824}]
I20260812 07:06:57.561430 4236 catalog_manager.cc:1347] Generated new cluster ID: f37eb8ecbc5244c79dcf4bce43646e56
I20260812 07:06:57.561510 4236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:06:57.592290 4236 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:06:57.593730 4236 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:06:57.607199 4236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e80b2b8ef2384386bb492373f1dcdea4: Generated new TSK 0
I20260812 07:06:57.608167 4236 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:06:57.644466 4135 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.641862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.27.110.252:32891)
0812 07:06:57.642035 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.642041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.643295 (+ 1254us) server_negotiation.cc:407] Connection header received
0812 07:06:57.643470 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.643478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.643538 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.643630 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.644328 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36528: BlockingRecv error: recv got EOF from 127.0.0.1:36528 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 07:06:57.646337 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.643025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.27.110.253:46165)
0812 07:06:57.643198 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.643202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.645275 (+ 2073us) server_negotiation.cc:407] Connection header received
0812 07:06:57.645523 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.645527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.645602 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.645662 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.646219 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58720: BlockingRecv error: recv got EOF from 127.0.0.1:58720 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260812 07:06:57.647996 3876 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.642552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.27.110.254:36275)
0812 07:06:57.642720 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.642725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.643522 (+ 797us) server_negotiation.cc:407] Connection header received
0812 07:06:57.647174 (+ 3652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.647178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.647234 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.647293 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.647877 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60964: BlockingRecv error: recv got EOF from 127.0.0.1:60964 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 07:06:57.653002 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.651462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.27.110.253:46165)
0812 07:06:57.651564 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.651568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.652087 (+ 519us) server_negotiation.cc:407] Connection header received
0812 07:06:57.652289 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.652293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.652359 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.652407 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.652909 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58732: BlockingRecv error: recv got EOF from 127.0.0.1:58732 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260812 07:06:57.657920 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.656598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.27.110.253:46165)
0812 07:06:57.656693 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.656697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.657121 (+ 424us) server_negotiation.cc:407] Connection header received
0812 07:06:57.657285 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.657287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.657332 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.657378 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.657830 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58746: BlockingRecv error: recv got EOF from 127.0.0.1:58746 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260812 07:06:57.663555 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.662083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.27.110.253:46165)
0812 07:06:57.662176 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.662180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.662672 (+ 492us) server_negotiation.cc:407] Connection header received
0812 07:06:57.662838 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.662841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.662888 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.662936 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.663464 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58762: BlockingRecv error: recv got EOF from 127.0.0.1:58762 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260812 07:06:57.671430 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.669899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.27.110.253:46165)
0812 07:06:57.670030 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.670034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.670557 (+ 523us) server_negotiation.cc:407] Connection header received
0812 07:06:57.670755 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.670758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.670803 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.670849 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.671336 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58764: BlockingRecv error: recv got EOF from 127.0.0.1:58764 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 07:06:57.682713 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.681565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58772 (local address 127.27.110.253:46165)
0812 07:06:57.681681 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.681685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.681921 (+ 236us) server_negotiation.cc:407] Connection header received
0812 07:06:57.682048 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.682051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.682106 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.682195 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.682608 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58772: BlockingRecv error: recv got EOF from 127.0.0.1:58772 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 07:06:57.701785 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.700697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58784 (local address 127.27.110.253:46165)
0812 07:06:57.700821 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.700825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.700985 (+ 160us) server_negotiation.cc:407] Connection header received
0812 07:06:57.701076 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.701080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.701137 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.701236 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.701669 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58784: BlockingRecv error: recv got EOF from 127.0.0.1:58784 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 07:06:57.737044 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.735924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.27.110.253:46165)
0812 07:06:57.736114 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.736118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.736233 (+ 115us) server_negotiation.cc:407] Connection header received
0812 07:06:57.736336 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.736338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.736386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.736459 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.736920 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58792: BlockingRecv error: recv got EOF from 127.0.0.1:58792 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260812 07:06:57.804558 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.803323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.27.110.253:46165)
0812 07:06:57.803491 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.803496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.803667 (+ 171us) server_negotiation.cc:407] Connection header received
0812 07:06:57.803814 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.803816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.803867 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.803943 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.804407 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58800: BlockingRecv error: recv got EOF from 127.0.0.1:58800 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 07:06:57.935846 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:57.934623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.27.110.253:46165)
0812 07:06:57.934783 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:57.934788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:57.934949 (+ 161us) server_negotiation.cc:407] Connection header received
0812 07:06:57.935111 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:57.935118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:57.935176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:57.935256 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:57.935723 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58810: BlockingRecv error: recv got EOF from 127.0.0.1:58810 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 07:06:58.195501 4000 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:06:58.194084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.27.110.253:46165)
0812 07:06:58.194254 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.194261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.194317 (+ 56us) server_negotiation.cc:407] Connection header received
0812 07:06:58.194517 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.194524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.194602 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.194718 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:06:58.195351 (+ 633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58818: BlockingRecv error: recv got EOF from 127.0.0.1:58818 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 07:06:58.471247 4192 catalog_manager.cc:1259] Loaded cluster ID: f37eb8ecbc5244c79dcf4bce43646e56
I20260812 07:06:58.471360 4192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: loading cluster ID for follower catalog manager: success
I20260812 07:06:58.473207 4192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: acquiring CA information for follower catalog manager: success
I20260812 07:06:58.474531 4192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d7ff020dd6b4c529d9618c34d0f6d72: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:06:58.488327 4226 catalog_manager.cc:1259] Loaded cluster ID: f37eb8ecbc5244c79dcf4bce43646e56
I20260812 07:06:58.488396 4226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: loading cluster ID for follower catalog manager: success
I20260812 07:06:58.490056 4226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: acquiring CA information for follower catalog manager: success
I20260812 07:06:58.491222 4226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a0d2f4086b594abea00752ff3fd3638a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:06:58.713462 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.709500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.27.110.253:46165)
0812 07:06:58.709795 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.709799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.709899 (+ 100us) server_negotiation.cc:407] Connection header received
0812 07:06:58.709994 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.709998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.710137 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.710245 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:58.710491 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.711065 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.711927 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.712155 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.712247 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.712523 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:58.712991 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:58.713089 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.713184 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.713241 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":92,"threads_started":1}
I20260812 07:06:58.720016 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.716592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.27.110.252:32891)
0812 07:06:58.716809 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.716813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.716841 (+ 28us) server_negotiation.cc:407] Connection header received
0812 07:06:58.716920 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.716922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.717032 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.717122 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:58.717311 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.717819 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.718577 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.718806 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.718881 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.719181 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:58.719585 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:58.719678 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.719769 (+ 91us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.719817 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260812 07:06:58.727665 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.723038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60972 (local address 127.27.110.254:36275)
0812 07:06:58.723674 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.723677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.723689 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:06:58.723768 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.723770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.723915 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.724038 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:58.724277 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.725067 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.726021 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.726299 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.726556 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.726878 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:58.727243 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:58.727325 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.727406 (+ 81us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.727487 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":130,"threads_started":1}
I20260812 07:06:58.727803 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.723318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58828 (local address 127.27.110.253:46165)
0812 07:06:58.723437 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.723441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.723927 (+ 486us) server_negotiation.cc:407] Connection header received
0812 07:06:58.724105 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.724108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.724162 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.724232 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:58.724613 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.725200 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.726480 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.726852 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.726952 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.727287 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:58.727377 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:58.727453 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.727610 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.727662 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:06:58.727988 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:06:58.723161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36542 (local address 127.27.110.252:32891)
0812 07:06:58.723511 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0812 07:06:58.723518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:06:58.723567 (+ 49us) server_negotiation.cc:407] Connection header received
0812 07:06:58.723736 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:06:58.723740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:06:58.723792 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:06:58.723872 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:06:58.724175 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.724984 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.726311 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:06:58.726641 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:06:58.726758 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:06:58.727052 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:06:58.727147 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:06:58.727234 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 07:06:58.727756 (+ 522us) server_negotiation.cc:299] Negotiation successful
0812 07:06:58.727813 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 07:06:58.731185 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3740
I20260812 07:06:58.751989 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 3879
I20260812 07:06:58.769496 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4004
2026-08-12T07:06:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3460 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 07:06:58 dist-test-slave-w4v5 krb5kdc[4416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 07:06:58 dist-test-slave-w4v5 krb5kdc[4416](info): set up 2 sockets
Aug 12 07:06:58 dist-test-slave-w4v5 krb5kdc[4416](info): commencing operation
krb5kdc: starting...
W20260812 07:07:00.072705 28091 mini_kdc.cc:121] Time spent starting KDC: real 1.273s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/test-user.keytab.
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T07:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T07:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
Entry for principal kudu/127.27.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab.
I20260812 07:07:00.202775 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36893
--webserver_interface=127.27.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36893,127.27.110.253:46497,127.27.110.252:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 07:07:00.316084 4435 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:07:00.316421 4435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:00.316499 4435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:00.316581 4435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:00.320487 4435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:00.320580 4435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:00.320618 4435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:00.320653 4435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:00.320681 4435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:00.325821 4435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36893,127.27.110.253:46497,127.27.110.252:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.254.keytab
--principal=kudu/127.27.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.254:36893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:07:00.326851 4435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:00.327876 4435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:00.337076 4445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:00.337100 4447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:00.337123 4443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:00.337321 4435 server_base.cc:1034] running on GCE node
I20260812 07:07:00.338153 4435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:00.339308 4435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:00.340478 4435 hybrid_clock.cc:648] HybridClock initialized: now 1786518420340449 us; error 41 us; skew 500 ppm
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:00.348894 4435 init.cc:377] Logged in from keytab as kudu/127.27.110.254@KRBTEST.COM (short username kudu)
I20260812 07:07:00.349767 4435 webserver.cc:460] Webserver started at http://127.27.110.254:36631/ using document root <none> and password file <none>
I20260812 07:07:00.350307 4435 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:00.350368 4435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:00.350548 4435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:00.352602 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data/instance:
uuid: "657fc2fbdc424a0094372cabd8d21260"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "dbd35e75c84f7828a889dbabb636ece1"
server_key_iv: "2869965442f5f7ca6c82aa7b1a57ca25"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.353119 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal/instance:
uuid: "657fc2fbdc424a0094372cabd8d21260"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "dbd35e75c84f7828a889dbabb636ece1"
server_key_iv: "2869965442f5f7ca6c82aa7b1a57ca25"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.356812 4435 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:07:00.359143 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:00.360412 4435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260812 07:07:00.360512 4435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal
uuid: "657fc2fbdc424a0094372cabd8d21260"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "dbd35e75c84f7828a889dbabb636ece1"
server_key_iv: "2869965442f5f7ca6c82aa7b1a57ca25"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.360636 4435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:00.381042 4435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:00.383898 4435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:00.384047 4435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:00.390957 4435 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.254:36893
I20260812 07:07:00.391296 4550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.254:36893 every 8 connection(s)
I20260812 07:07:00.391948 4435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/data/info.pb
I20260812 07:07:00.394598 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:00.398224 4554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.400180 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4435
I20260812 07:07:00.400328 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-0/wal/instance
I20260812 07:07:00.400628 28091 external_mini_cluster.cc:1428] Setting key f1f9745fe265520282a3f1819c1cc6cb
WARNING: no policy specified for kudu/127.27.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.253@KRBTEST.COM" created.
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:07:00.421516 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.400662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36893 (local address 127.0.0.1:42936)
0812 07:07:00.403982 (+ 3320us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.404036 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.405111 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.405986 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.405991 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.406338 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.406593 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.406605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.408442 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.408444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.409179 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.409187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.410076 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.410079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.410241 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.410866 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.410882 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.415967 (+ 5085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.418523 (+ 2556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.418532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.418540 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.418782 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.419202 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.419204 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.419206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.419318 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.420148 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.420153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.420274 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.420410 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.420675 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3098,"mutex_wait_us":44,"thread_start_us":126,"threads_started":1}
I20260812 07:07:00.421525 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.401519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42936 (local address 127.27.110.254:36893)
0812 07:07:00.401880 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.401897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.404137 (+ 2240us) server_negotiation.cc:407] Connection header received
0812 07:07:00.405488 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.405497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.405765 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.405971 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.406761 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.408311 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.409306 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.409923 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.410111 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.416158 (+ 6047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.416181 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.416193 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.416216 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.418397 (+ 2181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.418903 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.418912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.418917 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.419069 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.419424 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.419431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.419435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.420019 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.420162 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.420557 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.420749 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":246,"thread_start_us":76,"threads_started":1}
I20260812 07:07:00.424057 4554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.424819 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.424519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55086)
0812 07:07:00.424686 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.424766 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 07:07:00.425046 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
I20260812 07:07:00.425844 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.425639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55102)
0812 07:07:00.425754 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.425797 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 07:07:00.426051 4554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:46497: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab.
Entry for principal kudu/127.27.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab.
I20260812 07:07:00.432309 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:46497
--webserver_interface=127.27.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab
--principal=kudu/127.27.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36893,127.27.110.253:46497,127.27.110.252:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:00.475317 4554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } attempt: 1
I20260812 07:07:00.476183 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.475881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55118)
0812 07:07:00.476029 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.476104 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 07:07:00.477013 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.476781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55120)
0812 07:07:00.476899 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.476952 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 07:07:00.477166 4554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:46497: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
I20260812 07:07:00.545334 4554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } attempt: 2
I20260812 07:07:00.546362 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.545999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55128)
0812 07:07:00.546193 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.546268 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 07:07:00.547137 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.546871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55142)
0812 07:07:00.546992 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.547047 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260812 07:07:00.547276 4554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.253:46497: Network error: Client connection negotiation failed: client connection to 127.27.110.253:46497: connect: Connection refused (error 111)
W20260812 07:07:00.558332 4570 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:07:00.558655 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:00.558717 4570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:00.558796 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:00.562839 4570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:00.562924 4570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:00.562973 4570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:00.563009 4570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:00.563040 4570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:00.568552 4570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36893,127.27.110.253:46497,127.27.110.252:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.253.keytab
--principal=kudu/127.27.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.253:46497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:07:00.569625 4570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:00.570667 4570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:00.579044 4590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:00.579039 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:00.579039 4588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:00.579138 4570 server_base.cc:1034] running on GCE node
I20260812 07:07:00.579939 4570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:00.581007 4570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:00.582170 4570 hybrid_clock.cc:648] HybridClock initialized: now 1786518420582137 us; error 45 us; skew 500 ppm
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:00.585018 4570 init.cc:377] Logged in from keytab as kudu/127.27.110.253@KRBTEST.COM (short username kudu)
I20260812 07:07:00.585909 4570 webserver.cc:460] Webserver started at http://127.27.110.253:46151/ using document root <none> and password file <none>
I20260812 07:07:00.586453 4570 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:00.586546 4570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:00.586797 4570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:00.588840 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data/instance:
uuid: "f4ad5b59ee174bf99c027194ebccc62c"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "f5ddd19b381cd9aa7f14a2b3de8c3056"
server_key_iv: "25765bd385652b8884e1e43407ffc177"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.589411 4570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal/instance:
uuid: "f4ad5b59ee174bf99c027194ebccc62c"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "f5ddd19b381cd9aa7f14a2b3de8c3056"
server_key_iv: "25765bd385652b8884e1e43407ffc177"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.593137 4570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 07:07:00.595618 4602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:00.596959 4570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:00.597143 4570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal
uuid: "f4ad5b59ee174bf99c027194ebccc62c"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "f5ddd19b381cd9aa7f14a2b3de8c3056"
server_key_iv: "25765bd385652b8884e1e43407ffc177"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.597275 4570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:00.610816 4570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:00.613942 4570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:00.614161 4570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:00.621537 4570 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.253:46497
I20260812 07:07:00.621837 4688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.253:46497 every 8 connection(s)
I20260812 07:07:00.622556 4570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/data/info.pb
I20260812 07:07:00.625397 4691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:00.629143 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4570
I20260812 07:07:00.629281 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-1/wal/instance
I20260812 07:07:00.629249 4691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.629563 28091 external_mini_cluster.cc:1428] Setting key dff7fbb11236f380553e8899f4a61a7c
I20260812 07:07:00.638490 4554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } attempt: 3
WARNING: no policy specified for kudu/127.27.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.110.252@KRBTEST.COM" created.
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:07:00.652199 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.631396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.27.110.254:36893)
0812 07:07:00.631548 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.631552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.632626 (+ 1074us) server_negotiation.cc:407] Connection header received
0812 07:07:00.633882 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.633884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.633941 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.634027 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.635825 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.636354 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.638169 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.638406 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.638552 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.645899 (+ 7347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.645905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.645911 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.645940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.649215 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.649894 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.649901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.649906 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.650011 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.650559 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.650565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.650569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.650864 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.650994 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.651807 (+ 813us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.651958 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 07:07:00.653430 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.631900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36893 (local address 127.0.0.1:42952)
0812 07:07:00.632435 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.632489 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.633641 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.634239 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.634251 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.634824 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.635650 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.635673 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.636500 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.636506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.638029 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.638041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.638605 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.638610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.638816 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.639553 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.639588 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.645712 (+ 6124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.649339 (+ 3627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.649353 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.649368 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.649762 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.650163 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.650169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.650171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.650414 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.650993 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.651003 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.651471 (+ 468us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.651724 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.652183 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":146,"threads_started":1}
I20260812 07:07:00.655203 4691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.658031 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.638951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55148)
0812 07:07:00.639198 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.639212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.639297 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.640045 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.640049 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.640256 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.640356 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.640361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.641748 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.641754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.642680 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.642693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.643565 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.643570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.643679 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.644739 (+ 1060us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.644755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.646457 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.656025 (+ 9568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.656031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.656033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.656314 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.656690 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.656696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.656698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.656836 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.657367 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.657370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.657482 (+ 112us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.657624 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.657801 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260812 07:07:00.658371 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.639059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.27.110.253:46497)
0812 07:07:00.639525 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.639538 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.639562 (+ 24us) server_negotiation.cc:407] Connection header received
0812 07:07:00.639608 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.639613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.639813 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.640038 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.640465 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.641593 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.642832 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.643415 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.643534 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.646613 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.646630 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.646648 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.646682 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.655911 (+ 9229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.656425 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.656430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.656431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.656508 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.656961 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.656964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.656965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.657253 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.657401 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.657717 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.658183 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":100,"threads_started":1}
I20260812 07:07:00.661346 4554 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:07:00.662365 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.661925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54762)
0812 07:07:00.662198 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.662278 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260812 07:07:00.663445 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.662955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54764)
0812 07:07:00.663202 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.663258 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260812 07:07:00.663628 4554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:33279: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab.
Entry for principal kudu/127.27.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab.
I20260812 07:07:00.666934 28091 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskh57nMD/build/debug/bin/kudu
/tmp/dist-test-taskh57nMD/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:33279
--webserver_interface=127.27.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab
--principal=kudu/127.27.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.110.212:36125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.110.254:36893,127.27.110.253:46497,127.27.110.252:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 07:07:00.667616 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.655828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55156)
0812 07:07:00.656494 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.656508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.656597 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.656991 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.656996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.657241 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.657399 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.657409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.658463 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.658467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.659271 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.659281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.660021 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.660025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.660139 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.661213 (+ 1074us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.661228 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.663171 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.665874 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.665880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.665886 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.666188 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.666518 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.666520 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.666522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.666600 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.667118 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.667122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.667247 (+ 125us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.667359 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.667499 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589}
I20260812 07:07:00.667904 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.655907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.27.110.253:46497)
0812 07:07:00.656277 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.656286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.656583 (+ 297us) server_negotiation.cc:407] Connection header received
0812 07:07:00.656718 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.656721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.656775 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.656952 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.657539 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.658325 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.659405 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.659875 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.660016 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.663342 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.663346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.663349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.663372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.665738 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.666308 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.666312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.666314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.666417 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.666702 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.666708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.666713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.666971 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.667113 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.667443 (+ 330us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.667599 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":151,"threads_started":1}
I20260812 07:07:00.668680 4691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.669423 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.669159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54778)
0812 07:07:00.669273 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.669367 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 07:07:00.669724 4603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
I20260812 07:07:00.670706 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.670497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54786)
0812 07:07:00.670606 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.670655 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260812 07:07:00.670867 4691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:33279: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
I20260812 07:07:00.694802 4554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } attempt: 1
I20260812 07:07:00.695744 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.695383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54792)
0812 07:07:00.695562 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.695647 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 07:07:00.696550 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.696311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54798)
0812 07:07:00.696431 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.696487 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 07:07:00.696748 4554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:33279: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
I20260812 07:07:00.720170 4691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } attempt: 1
I20260812 07:07:00.721119 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.720812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54810)
0812 07:07:00.720970 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.721037 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 07:07:00.721897 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.721672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54818)
0812 07:07:00.721784 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.721839 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 07:07:00.722029 4691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:33279: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
I20260812 07:07:00.771924 4554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } attempt: 2
I20260812 07:07:00.772841 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.772456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54824)
0812 07:07:00.772645 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.772735 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 07:07:00.773630 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.773368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54834)
0812 07:07:00.773495 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.773555 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 07:07:00.773815 4554 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:33279: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
I20260812 07:07:00.790226 4691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } attempt: 2
I20260812 07:07:00.791230 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.790885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54848)
0812 07:07:00.791038 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.791140 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 07:07:00.791983 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.791764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54860)
0812 07:07:00.791876 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.791923 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 07:07:00.792128 4691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.110.252:33279: Network error: Client connection negotiation failed: client connection to 127.27.110.252:33279: connect: Connection refused (error 111)
W20260812 07:07:00.793532 4702 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 07:07:00.793893 4702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 07:07:00.793960 4702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 07:07:00.794047 4702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 07:07:00.798743 4702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 07:07:00.798834 4702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 07:07:00.798882 4702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 07:07:00.798949 4702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 07:07:00.798996 4702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 07:07:00.804896 4702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.110.212:36125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.110.254:36893,127.27.110.253:46497,127.27.110.252:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/krb5kdc/kudu_127.27.110.252.keytab
--principal=kudu/127.27.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.110.252:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 12 Aug 2026 06:15:54 UTC on ccycloud.quasar-rnfzce.root.comops.site
build id 4581989
I20260812 07:07:00.806537 4702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 07:07:00.807703 4702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 07:07:00.817235 4719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:00.817142 4722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 07:07:00.817169 4715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 07:07:00.817354 4702 server_base.cc:1034] running on GCE node
I20260812 07:07:00.818176 4702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 07:07:00.819410 4702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 07:07:00.820608 4702 hybrid_clock.cc:648] HybridClock initialized: now 1786518420820573 us; error 59 us; skew 500 ppm
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 07:07:00.823616 4702 init.cc:377] Logged in from keytab as kudu/127.27.110.252@KRBTEST.COM (short username kudu)
I20260812 07:07:00.824625 4702 webserver.cc:460] Webserver started at http://127.27.110.252:35883/ using document root <none> and password file <none>
I20260812 07:07:00.825212 4702 fs_manager.cc:359] Metadata directory not provided
I20260812 07:07:00.825309 4702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 07:07:00.825560 4702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 07:07:00.827772 4702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data/instance:
uuid: "adeebe54435c4191b9fe5eb0a3cc6327"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "07a4a57b12e411a05e42f729c4683411"
server_key_iv: "085b2b8add6ff4852fec6c360e276f2d"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.828346 4702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal/instance:
uuid: "adeebe54435c4191b9fe5eb0a3cc6327"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "07a4a57b12e411a05e42f729c4683411"
server_key_iv: "085b2b8add6ff4852fec6c360e276f2d"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.832358 4702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 07:07:00.834851 4738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 07:07:00.836298 4702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 07:07:00.836448 4702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data,/tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal
uuid: "adeebe54435c4191b9fe5eb0a3cc6327"
format_stamp: "Formatted at 2026-08-12 07:07:00 on dist-test-slave-w4v5"
server_key: "07a4a57b12e411a05e42f729c4683411"
server_key_iv: "085b2b8add6ff4852fec6c360e276f2d"
server_key_version: "encryptionkey@0"
I20260812 07:07:00.836568 4702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 07:07:00.858937 4702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 07:07:00.862355 4702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 07:07:00.862571 4702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 07:07:00.870220 4702 rpc_server.cc:307] RPC server started. Bound to: 127.27.110.252:33279
I20260812 07:07:00.870550 4835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.110.252:33279 every 8 connection(s)
I20260812 07:07:00.871330 4702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/data/info.pb
I20260812 07:07:00.873046 4554 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } attempt: 3
I20260812 07:07:00.874756 4838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 07:07:00.876106 28091 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskh57nMD/build/debug/bin/kudu as pid 4702
I20260812 07:07:00.876255 28091 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskh57nMD/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786518006756923-28091-0/minicluster-data/master-2/wal/instance
I20260812 07:07:00.876611 28091 external_mini_cluster.cc:1428] Setting key 2d8e8f5138ce3b8a7468dd03ee421e3b
I20260812 07:07:00.880431 4838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.254@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:07:00.883276 4691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } attempt: 3
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:07:00.888270 4560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.873649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54862)
0812 07:07:00.873823 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.873842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.873951 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.876592 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.876595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.876815 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.876932 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.876940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.878652 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.878657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.879326 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.879337 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.880485 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.880490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.880583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.881185 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.881201 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.882929 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.886008 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.886014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.886017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.886294 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.886726 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.886729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.886731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.886841 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.887741 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.887744 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.887841 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.887919 (+ 78us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.888055 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 07:07:00.890223 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.874485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.27.110.252:33279)
0812 07:07:00.875012 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.875028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.875065 (+ 37us) server_negotiation.cc:407] Connection header received
0812 07:07:00.875823 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.875842 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.876257 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.876650 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.877100 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.878516 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.879458 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.880284 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.880476 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.883176 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.883182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.883198 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.883229 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.885877 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.886445 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.886451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.886458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.886606 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.886953 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.886957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.886958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.887609 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.888627 (+ 1018us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.888807 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.889155 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":372,"thread_start_us":167,"threads_started":1}
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.253@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:07:00.890764 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.877247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.27.110.252:33279)
0812 07:07:00.877543 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.877552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.877809 (+ 257us) server_negotiation.cc:407] Connection header received
0812 07:07:00.877945 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.877949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.877991 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.878092 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.879162 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.879784 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.881025 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.881420 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.881537 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.885435 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.885441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.885444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.885472 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.887793 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.888625 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.888629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.888634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.888753 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.889155 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.889158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.889160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.889350 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.890265 (+ 915us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.890357 (+ 92us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.890498 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"spinlock_wait_cycles":7040,"thread_start_us":111,"threads_started":1}
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:07:00.895889 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260: Bootstrap starting.
I20260812 07:07:00.895953 4694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.883724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54894)
0812 07:07:00.883898 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.883915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.884000 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.884790 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.884796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.885014 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.885146 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.885156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.886569 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.886576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.887611 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.887628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.888396 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.888402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.888501 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.889218 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.889239 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.890938 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.893952 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.893956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.893959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.894276 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.894612 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.894617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.894619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.894740 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.895304 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.895308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.895437 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.895584 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.895752 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 07:07:00.896518 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.883954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54894 (local address 127.27.110.252:33279)
0812 07:07:00.884488 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.884498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.884513 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:07:00.884548 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.884551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.884595 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.885345 (+ 750us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.885448 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.886435 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.887774 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.888236 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.888721 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.891135 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.891138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.891142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.891177 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.893810 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.894414 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.894417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.894419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.894502 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.894868 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.894871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.894873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.895149 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.895249 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.896188 (+ 939us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.896312 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":456,"thread_start_us":122,"threads_started":1}
I20260812 07:07:00.898885 4554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:00.899775 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.882936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.27.110.254:36893)
0812 07:07:00.883100 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.883108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.883601 (+ 493us) server_negotiation.cc:407] Connection header received
0812 07:07:00.884073 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.884080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.884147 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.884238 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.885683 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.886951 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.888242 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.888570 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.888649 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.893256 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.893261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.893264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.893290 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.894991 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.895610 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.895613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.895615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.895702 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.896044 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.896047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.896049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.896287 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.896372 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.899439 (+ 3067us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.899571 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 07:07:00.899817 4691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: Bootstrap starting.
I20260812 07:07:00.899885 4554 log.cc:824] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:00.900930 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.882653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.254:36893 (local address 127.0.0.1:42966)
0812 07:07:00.883385 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.883498 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.883641 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.884298 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.884306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.885181 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.885512 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.885523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.887080 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.887083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.888114 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.888125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.889537 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.889545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.889689 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.891102 (+ 1413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.891119 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.893105 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.895149 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.895159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.895165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.895470 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.895815 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.895818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.895820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.895911 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.896443 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.896448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.896579 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.900343 (+ 3764us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.900675 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":69,"thread_start_us":112,"threads_started":1}
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.254@KRBTEST.COM
I20260812 07:07:00.902138 4554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260: No bootstrap required, opened a new log
I20260812 07:07:00.902544 4838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.902819 4691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:00.904013 4691 log.cc:824] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:00.906337 4691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: No bootstrap required, opened a new log
I20260812 07:07:00.906534 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.893893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42982 (local address 127.27.110.254:36893)
0812 07:07:00.894817 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.894834 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.894850 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:07:00.894886 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.894889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.894951 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.895942 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.896503 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.897346 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.898220 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.898514 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.898632 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.901970 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.901974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.901977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.902006 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.904489 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.905086 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.905090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.905091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.905191 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.905512 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.905515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.905517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.905744 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.905969 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.906150 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.906275 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":101,"threads_started":1}
I20260812 07:07:00.906509 4554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.907192 4554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657fc2fbdc424a0094372cabd8d21260, State: Initialized, Role: FOLLOWER
I20260812 07:07:00.907555 4554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.908967 4554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:00.911360 4691 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.911613 4691 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ad5b59ee174bf99c027194ebccc62c, State: Initialized, Role: FOLLOWER
I20260812 07:07:00.911937 4691 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.912009 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.912140 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: This master's current role is: FOLLOWER
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:07:00.912607 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.912751 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:00.913282 4691 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.110.253@KRBTEST.COM
I20260812 07:07:00.917512 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.903419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.253:46497 (local address 127.0.0.1:55170)
0812 07:07:00.903547 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.903564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.903643 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.907033 (+ 3390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.907037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.907258 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.907399 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.907406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.908581 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.908585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.909444 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.909458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.910149 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.910153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.910238 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.910835 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.910850 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.912829 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.915659 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.915663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.915665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.915939 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.916284 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.916290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.916292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.916385 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.916917 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.916920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.917040 (+ 120us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.917146 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.917301 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 07:07:00.918147 4701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.903277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.27.110.253:46497)
0812 07:07:00.906784 (+ 3507us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.906787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.906799 (+ 12us) server_negotiation.cc:407] Connection header received
0812 07:07:00.906872 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.906875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.906913 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.907013 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.907532 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.908284 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.909588 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.909926 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.909999 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.912975 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.912981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.912984 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.913008 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.915509 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.916078 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.916081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.916084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.916172 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.916505 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.916508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.916510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.916785 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.916870 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.917847 (+ 977us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.917977 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3427}
I20260812 07:07:00.918912 4838 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
I20260812 07:07:00.914698 4435 master_runner.cc:186] Error getting master registration for 127.27.110.253:46497: OK, instance_id { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" instance_seqno: 1786518420619583 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 07:07:00.919593 4435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 07:07:00.920929 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.908093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.27.110.253:46497)
0812 07:07:00.908192 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.908196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.908343 (+ 147us) server_negotiation.cc:407] Connection header received
0812 07:07:00.908531 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.908540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.908590 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.908660 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.909547 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.910258 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.911466 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.911872 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.911952 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.916817 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.916823 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.916828 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.916853 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.918500 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.919402 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.919405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.919407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.919495 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.919920 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.919923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.919925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.920131 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.920321 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.920685 (+ 364us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.920799 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260812 07:07:00.923143 4877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:07:00.923266 4877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:07:00.921278 4570 master_runner.cc:186] Error getting master registration for 127.27.110.252:33279: OK, instance_id { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" instance_seqno: 1786518420868243 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260812 07:07:00.924476 4900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:07:00.924530 4900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:07:00.924546 4570 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 12 07:07:00 dist-test-slave-w4v5 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786518420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.110.252@KRBTEST.COM for kudu/127.27.110.252@KRBTEST.COM
I20260812 07:07:00.928176 28091 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 07:07:00.928866 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.919422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.110.252:33279 (local address 127.0.0.1:54896)
0812 07:07:00.919671 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 07:07:00.919686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 07:07:00.919755 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 07:07:00.920481 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 07:07:00.920484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 07:07:00.920668 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0812 07:07:00.920754 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.920760 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.921748 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.921752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.922494 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 07:07:00.922506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.923226 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.923231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 07:07:00.923324 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.923829 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 07:07:00.923843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0812 07:07:00.925276 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 07:07:00.927236 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.927240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.927243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.927499 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.927775 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 07:07:00.927778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 07:07:00.927779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0812 07:07:00.927856 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 07:07:00.928352 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 07:07:00.928355 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 07:07:00.928444 (+ 89us) client_negotiation.cc:769] Sending connection context
0812 07:07:00.928580 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 07:07:00.928712 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260812 07:07:00.929409 4499 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "657fc2fbdc424a0094372cabd8d21260"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42982
I20260812 07:07:00.929383 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.920098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.27.110.252:33279)
0812 07:07:00.920204 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.920208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.920221 (+ 13us) server_negotiation.cc:407] Connection header received
0812 07:07:00.920253 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.920256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.920297 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.920372 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.920868 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.921613 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.922641 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.922968 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.923065 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.925450 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.925453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.925455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.925487 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.927064 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.927604 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.927607 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.927609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.927687 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.927964 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.927967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.927969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.928179 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.928271 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.929110 (+ 839us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.929248 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260812 07:07:00.929661 4499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 07:07:00.929770 4499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:00.931291 4499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.933194 4646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "657fc2fbdc424a0094372cabd8d21260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f4ad5b59ee174bf99c027194ebccc62c"
I20260812 07:07:00.933439 4646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:00.934165 4838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: Bootstrap starting.
I20260812 07:07:00.934782 4646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 657fc2fbdc424a0094372cabd8d21260 in term 1.
I20260812 07:07:00.935256 4461 leader_election.cc:302] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 657fc2fbdc424a0094372cabd8d21260, f4ad5b59ee174bf99c027194ebccc62c; no voters:
I20260812 07:07:00.935573 4856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 07:07:00.936221 4856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [term 1 LEADER]: Becoming Leader. State: Replica: 657fc2fbdc424a0094372cabd8d21260, State: Running, Role: LEADER
I20260812 07:07:00.936582 4856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.937044 4838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: Neither blocks nor log segments found. Creating new log.
I20260812 07:07:00.937956 4838 log.cc:824] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: Log is configured to *not* fsync() on all Append() calls
I20260812 07:07:00.938457 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 657fc2fbdc424a0094372cabd8d21260. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.938582 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:00.939043 4911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 07:07:00.939278 4499 leader_election.cc:288] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [CANDIDATE]: Term 1 election: Requested vote from peers f4ad5b59ee174bf99c027194ebccc62c (127.27.110.253:46497), adeebe54435c4191b9fe5eb0a3cc6327 (127.27.110.252:33279)
I20260812 07:07:00.940280 4911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 07:07:00.953945 4838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: No bootstrap required, opened a new log
I20260812 07:07:00.957583 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.941953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42994 (local address 127.27.110.254:36893)
0812 07:07:00.942070 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.942076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.945302 (+ 3226us) server_negotiation.cc:407] Connection header received
0812 07:07:00.945599 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.945603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.945655 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.945707 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.947067 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.947918 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.949759 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.950122 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.950201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.953000 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.953005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.953008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.953038 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.954850 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.955731 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.955735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.955737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.955830 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.956317 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.956321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.956323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.956571 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.956690 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.957256 (+ 566us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.957398 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260812 07:07:00.959134 4838 raft_consensus.cc:348] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.959414 4838 raft_consensus.cc:729] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adeebe54435c4191b9fe5eb0a3cc6327, State: Initialized, Role: FOLLOWER
I20260812 07:07:00.959789 4838 consensus_queue.cc:260] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } }
I20260812 07:07:00.961290 4838 sys_catalog.cc:564] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 07:07:00.961352 4786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "657fc2fbdc424a0094372cabd8d21260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "adeebe54435c4191b9fe5eb0a3cc6327"
I20260812 07:07:00.961624 4786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [term 0 FOLLOWER]: Advancing to term 1
I20260812 07:07:00.963215 4786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 657fc2fbdc424a0094372cabd8d21260 in term 1.
I20260812 07:07:00.963335 4924 sys_catalog.cc:455] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.963455 4924 sys_catalog.cc:458] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:00.969707 4695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.942938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.27.110.253:46497)
0812 07:07:00.943061 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.943066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.945279 (+ 2213us) server_negotiation.cc:407] Connection header received
0812 07:07:00.945487 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.945490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.945552 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.945605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.948130 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.948933 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.963264 (+ 14331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.963712 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.963804 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.965990 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.965994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.965997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.966029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.967800 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.968292 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.968296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.968297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.968397 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.968710 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.968713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.968715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.968931 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.969042 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.969391 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.969537 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260812 07:07:00.971552 4850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:00.944683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.27.110.252:33279)
0812 07:07:00.944798 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:00.944802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:00.946209 (+ 1407us) server_negotiation.cc:407] Connection header received
0812 07:07:00.946368 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:00.946371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:00.946415 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:00.946463 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:00.950248 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.951028 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.964496 (+ 13468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:00.964850 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:00.964931 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:00.967803 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 07:07:00.967807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 07:07:00.967810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 07:07:00.967835 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 07:07:00.969516 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.970106 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.970109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.970111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.970197 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 07:07:00.970563 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 07:07:00.970566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 07:07:00.970568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 07:07:00.970778 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 07:07:00.970912 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:00.971269 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 07:07:00.971385 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260812 07:07:00.977517 4786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [term 1 FOLLOWER]: Refusing update from remote peer 657fc2fbdc424a0094372cabd8d21260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 07:07:00.977744 4646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [term 1 FOLLOWER]: Refusing update from remote peer 657fc2fbdc424a0094372cabd8d21260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260812 07:07:00.978385 4945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 07:07:00.978456 4945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 07:07:00.978364 4904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:00.979306 4904 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 07:07:00.982932 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 657fc2fbdc424a0094372cabd8d21260. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.983062 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:00.986887 4924 sys_catalog.cc:455] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 657fc2fbdc424a0094372cabd8d21260. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.987063 4924 sys_catalog.cc:458] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:00.989403 4904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.989444 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.989528 4904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:00.989565 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260 [sys.catalog]: This master's current role is: LEADER
I20260812 07:07:00.991297 4911 catalog_manager.cc:1347] Generated new cluster ID: 5e206b655b834fc4ac34645ab45213b7
I20260812 07:07:00.991360 4911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 07:07:00.992050 4924 sys_catalog.cc:455] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.992170 4924 sys_catalog.cc:458] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:00.993784 4953 mvcc.cc:205] Tried to move back new op lower bound from 7317579452312506368 to 7317579452159950848. Current Snapshot: MvccSnapshot[applied={T|T < 7317579452312506368}]
I20260812 07:07:00.994019 4919 mvcc.cc:205] Tried to move back new op lower bound from 7317579452312506368 to 7317579452159950848. Current Snapshot: MvccSnapshot[applied={T|T < 7317579452312506368 or (T in {7317579452312506368})}]
I20260812 07:07:00.996068 4948 mvcc.cc:205] Tried to move back new op lower bound from 7317579452312506368 to 7317579452159950848. Current Snapshot: MvccSnapshot[applied={T|T < 7317579452312506368}]
I20260812 07:07:00.996238 4881 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "657fc2fbdc424a0094372cabd8d21260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "657fc2fbdc424a0094372cabd8d21260" member_type: VOTER last_known_addr { host: "127.27.110.254" port: 36893 } } peers { permanent_uuid: "f4ad5b59ee174bf99c027194ebccc62c" member_type: VOTER last_known_addr { host: "127.27.110.253" port: 46497 } } peers { permanent_uuid: "adeebe54435c4191b9fe5eb0a3cc6327" member_type: VOTER last_known_addr { host: "127.27.110.252" port: 33279 } } }
I20260812 07:07:00.996346 4881 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c [sys.catalog]: This master's current role is: FOLLOWER
I20260812 07:07:01.006306 4911 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 07:07:01.007594 4911 catalog_manager.cc:1504] Loading token signing keys...
I20260812 07:07:01.024935 4911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 657fc2fbdc424a0094372cabd8d21260: Generated new TSK 0
I20260812 07:07:01.025882 4911 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 07:07:01.067404 4851 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.064676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.27.110.254:36893)
0812 07:07:01.064822 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.064827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.066025 (+ 1198us) server_negotiation.cc:407] Connection header received
0812 07:07:01.066483 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.066487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.066542 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.066596 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.067270 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42996: BlockingRecv error: recv got EOF from 127.0.0.1:42996 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260812 07:07:01.068760 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.065326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.27.110.253:46497)
0812 07:07:01.065447 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.065451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.066379 (+ 928us) server_negotiation.cc:407] Connection header received
0812 07:07:01.067951 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.067954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.068006 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.068059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.068641 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55196: BlockingRecv error: recv got EOF from 127.0.0.1:55196 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260812 07:07:01.071105 4850 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.069690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.27.110.252:33279)
0812 07:07:01.069807 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.069812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.070124 (+ 312us) server_negotiation.cc:407] Connection header received
0812 07:07:01.070294 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.070298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.070349 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.070400 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.070947 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54904: BlockingRecv error: recv got EOF from 127.0.0.1:54904 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260812 07:07:01.075810 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.074160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.27.110.253:46497)
0812 07:07:01.074263 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.074267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.074778 (+ 511us) server_negotiation.cc:407] Connection header received
0812 07:07:01.074987 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.074991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.075066 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.075138 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.075697 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55200: BlockingRecv error: recv got EOF from 127.0.0.1:55200 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260812 07:07:01.081133 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.079407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.27.110.253:46497)
0812 07:07:01.079576 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.079580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.080140 (+ 560us) server_negotiation.cc:407] Connection header received
0812 07:07:01.080334 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.080338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.080389 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.080443 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.081001 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55212: BlockingRecv error: recv got EOF from 127.0.0.1:55212 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260812 07:07:01.087200 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.085448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.27.110.253:46497)
0812 07:07:01.085586 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.085590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.086160 (+ 570us) server_negotiation.cc:407] Connection header received
0812 07:07:01.086370 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.086374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.086428 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.086495 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.087059 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55222: BlockingRecv error: recv got EOF from 127.0.0.1:55222 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260812 07:07:01.095218 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.093515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.27.110.253:46497)
0812 07:07:01.093659 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.093663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.094213 (+ 550us) server_negotiation.cc:407] Connection header received
0812 07:07:01.094409 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.094412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.094479 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.094531 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.095103 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55236: BlockingRecv error: recv got EOF from 127.0.0.1:55236 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 07:07:01.107795 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.105793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55238 (local address 127.27.110.253:46497)
0812 07:07:01.105924 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.105929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.106574 (+ 645us) server_negotiation.cc:407] Connection header received
0812 07:07:01.106812 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.106816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.106891 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.106954 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.107671 (+ 717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55238: BlockingRecv error: recv got EOF from 127.0.0.1:55238 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 07:07:01.128525 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.126697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.27.110.253:46497)
0812 07:07:01.126829 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.126836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.127508 (+ 672us) server_negotiation.cc:407] Connection header received
0812 07:07:01.127720 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.127723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.127777 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.127828 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.128383 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55248: BlockingRecv error: recv got EOF from 127.0.0.1:55248 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260812 07:07:01.164853 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.163444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.27.110.253:46497)
0812 07:07:01.163589 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.163596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.163810 (+ 214us) server_negotiation.cc:407] Connection header received
0812 07:07:01.163969 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.163972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.164024 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.164135 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.164716 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260812 07:07:01.232425 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.231032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.27.110.253:46497)
0812 07:07:01.231231 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.231236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.231417 (+ 181us) server_negotiation.cc:407] Connection header received
0812 07:07:01.231621 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.231624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.231715 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.231817 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.232298 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55272: BlockingRecv error: recv got EOF from 127.0.0.1:55272 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260812 07:07:01.364291 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.362836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.27.110.253:46497)
0812 07:07:01.363003 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.363009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.363259 (+ 250us) server_negotiation.cc:407] Connection header received
0812 07:07:01.363474 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.363477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.363528 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.363668 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.364152 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55286: BlockingRecv error: recv got EOF from 127.0.0.1:55286 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 07:07:01.624032 4695 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 07:07:01.622688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.27.110.253:46497)
0812 07:07:01.622853 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:01.622857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:01.623038 (+ 181us) server_negotiation.cc:407] Connection header received
0812 07:07:01.623226 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:01.623228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:01.623280 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:01.623354 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 07:07:01.623916 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55298: BlockingRecv error: recv got EOF from 127.0.0.1:55298 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 07:07:01.925917 4900 catalog_manager.cc:1259] Loaded cluster ID: 5e206b655b834fc4ac34645ab45213b7
I20260812 07:07:01.926016 4900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: loading cluster ID for follower catalog manager: success
I20260812 07:07:01.927827 4900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: acquiring CA information for follower catalog manager: success
I20260812 07:07:01.929059 4900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4ad5b59ee174bf99c027194ebccc62c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:07:01.979857 4945 catalog_manager.cc:1259] Loaded cluster ID: 5e206b655b834fc4ac34645ab45213b7
I20260812 07:07:01.979943 4945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: loading cluster ID for follower catalog manager: success
I20260812 07:07:01.981570 4945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: acquiring CA information for follower catalog manager: success
I20260812 07:07:01.982734 4945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P adeebe54435c4191b9fe5eb0a3cc6327: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 07:07:02.142869 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.138674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.27.110.253:46497)
0812 07:07:02.138970 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:02.138974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:02.138994 (+ 20us) server_negotiation.cc:407] Connection header received
0812 07:07:02.139161 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:02.139164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:02.139348 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:02.139451 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:02.139779 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.140397 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.141360 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.141623 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.141706 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.141990 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:02.142432 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:02.142565 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:02.142613 (+ 48us) server_negotiation.cc:299] Negotiation successful
0812 07:07:02.142667 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260812 07:07:02.149618 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.146026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54910 (local address 127.27.110.252:33279)
0812 07:07:02.146272 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:02.146276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:02.146291 (+ 15us) server_negotiation.cc:407] Connection header received
0812 07:07:02.146371 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:02.146373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:02.146487 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:02.146591 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:02.146796 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.147372 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.148154 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.148375 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.148457 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.148799 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:02.149230 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:02.149326 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:02.149430 (+ 104us) server_negotiation.cc:299] Negotiation successful
0812 07:07:02.149484 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260812 07:07:02.157078 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.153198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.27.110.253:46497)
0812 07:07:02.153334 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:02.153338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:02.153514 (+ 176us) server_negotiation.cc:407] Connection header received
0812 07:07:02.153683 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:02.153685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:02.153732 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:02.153818 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:02.154064 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.154740 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.155831 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.156140 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.156250 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.156581 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:02.156670 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:02.156752 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:02.156874 (+ 122us) server_negotiation.cc:299] Negotiation successful
0812 07:07:02.156932 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260812 07:07:02.157694 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.152960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.27.110.252:33279)
0812 07:07:02.153084 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:02.153106 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:02.153871 (+ 765us) server_negotiation.cc:407] Connection header received
0812 07:07:02.154059 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:02.154063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:02.154127 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:02.154199 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:02.154563 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.155313 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.156384 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.156707 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.156803 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.157185 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:02.157265 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:02.157319 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:02.157476 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 07:07:02.157527 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 07:07:02.157873 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 07:07:02.152848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.27.110.254:36893)
0812 07:07:02.153541 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0812 07:07:02.153546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 07:07:02.153562 (+ 16us) server_negotiation.cc:407] Connection header received
0812 07:07:02.153666 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 07:07:02.153671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 07:07:02.153858 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 07:07:02.153986 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 07:07:02.154213 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.154870 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.155972 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 07:07:02.156321 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 07:07:02.156430 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 07:07:02.156763 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 07:07:02.157110 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 07:07:02.157223 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 07:07:02.157676 (+ 453us) server_negotiation.cc:299] Negotiation successful
0812 07:07:02.157736 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":96,"threads_started":1}
I20260812 07:07:02.165091 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4435
I20260812 07:07:02.184557 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4570
I20260812 07:07:02.197510 28091 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskh57nMD/build/debug/bin/kudu with pid 4702
2026-08-12T07:07:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3428 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6889 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (415411 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS